[D] Training code and more released for “The Era of 1 Bit LLMs”

Unfortunately Microsoft didn’t release the weights but if it’s between that and the training code then this is certainly preferable. [The post is here](https://github.com/microsoft/unilm/blob/master/bitnet/The-Era-of-1-bit-LLMs__Training_Tips_Code_FAQ.pdf).

What do we think? Any ideas about that oddly shaped loss curve? Do you think this approach breaks after 4B parameters? Thoughts on how LLMs could even work at such a low precision? I know it’s not particularly scientific but it seems pretty counterintuitive to me that you could potentially fit a fully functional 7B parameter …

functional ideas llms loss low machinelearning parameters precision scientific think thoughts work