[P] Silly project: implement MLP using a transformer (yo, dawg...)

Transformers heavily rely on MLPs. Presumably, facts which LLMs can recall are stored in MLPs. (E.g. see ROME paper.) These MLPs are huge.

E.g. consider GPT-Neo-350M. Each MLP has 1024-element input and output layers and 4096-element inner layer. This requires 2x1024x4096 = 4.2M weights. Whole model has 24 of them (one per layer), resulting in 100M weights being used for MLPs.

And yet these MLPs basically just do 4096 dot products to calculate features. Millions of weights just to calculate …

element facts gpt gpt-neo layer llms machinelearning mlp neo paper project recall them transformer transformers