[N] Introducing DBRX: A New Standard for Open LLM

[https://x.com/vitaliychiley/status/1772958872891752868?s=20](https://x.com/vitaliychiley/status/1772958872891752868?s=20)

Shill disclaimer: I was the pretraining lead for the project

DBRX deets:

* 16 Experts (12B params per single expert; top\_k=4 routing)
* 36B active params (132B total params)
* trained for 12T tokens
* 32k sequence length training

expert experts machinelearning params per pretraining project routing tokens total training