[N] Live and open training of BigScience's 176B multilingual language model has just started

The \[BigScience project\]([https://bigscience.huggingface.co](https://bigscience.huggingface.co)) has just started the training of its main model and the training can be **followed live** here: [https://twitter.com/BigScienceLLM](https://twitter.com/BigScienceLLM) and here: [https://huggingface.co/bigscience/tr11-176B-ml-logs/tensorboard#scalars&tagFilter=loss](https://huggingface.co/bigscience/tr11-176B-ml-logs/tensorboard#scalars&tagFilter=loss)

Here are more information on the model, dataset, engineering, training and hardware:

1. **The model**:

* 176B parameters decoder-only architecture (GPT-like)
* 70 layers - 112 attention heads per layers - hidden dimensionality of 14336 - 2048 tokens sequence length
* ALiBi positional embeddings - GeLU activation function
* **Read more**:
* Blog post summarizing how …

bigscience language language model machinelearning training