[D] Nemotron-4 340b detailed analysis

I took a look at NVIDIA's 340B Nemotron LLM - some of my findings:

* **Squared ReLU** unlike Llama SwiGLU, Gemma GeGLU. Different to GLU variants found in [arxiv.org/pdf/2002.05202](http://arxiv.org/pdf/2002.05202) (GLU Variants Improve Transformer, Noam Shazeer)
* ReGLU is \[ ReLU(X \* W\_gate) \* (X \* W\_up) \] \* W\_down
* We need 2 ReLUs + tied weights \[ ReLU(X \* W\_up) \* ReLU(X \* W\_up) \] \* W\_down, so bit like GLU, but not the same
* Why does Squared …

analysis llm look machinelearning nvidia relu