[P] Classification finetuning experiments on small GPT-2 sized LLMs

I ran a few classification finetuning experiments on relatively "small" experiments that I found interesting and wanted to share:

|Model|Weights|Trainable token|Trainable layers|Context length|CPU/GPU|Training time|Training acc|Validation acc|Test acc|
|:-|:-|:-|:-|:-|:-|:-|:-|:-|:-|
|1|gpt2-small (124M)|pretrained|last|last\_block|longest train ex. (120)|V100|0.39 min|96.63%|97.99%|
|2|gpt2-small (124M)|pretrained|first|last\_block|longest train ex. (120)|V100|0.37 min|78.46%|80.54%|
|3|gpt2-small (124M)|pretrained|last|last\_layer|longest train ex. (120)|V100|0.33 min|78.65%|87.25%|
|4|gpt2-small (124M)|pretrained|last|all|longest train ex. (120)|V100|0.94 min|99.62%|96.64%|
|5|gpt2-medium (355M)|pretrained|last|last\_block|longest train ex. (120)|V100|0.91 min|87.50%|51.01%|
|6|gpt2-large (774M)|pretrained|last|last\_block|longest train ex. (120)|V100|1.91 min|99.52%|98.66%|
|7|gpt2-small (124M)|random|last|all|longest train ex. (120)|V100|0.93 min|100%|97.32%|
|8|gpt2-small (124M)|pretrained|last|last\_block|context length (1024)|V100|3.24 min|83.08%|87.92%|

1. Training the Last vs. …

acc classification context cpu finetuning found gpt gpt-2 gpu llms machinelearning min ran small test token train training v100 validation