[D] Initializing a Small LLM to Reflect Natural Token Distribution

Hello !

​

Is it feasible to set up the model's weights in such a way that the output of the final softmax layer, prior to any training, mirrors the distribution of tokens in the training data?

My initial thought is to initialize all weights and biases to zero, and then modify the softmax layer (which would initially output zeros) by incorporating a pre-calculated vector of observed token probabilities. I haven't come across this approach in my research thus far …

biases data distribution hello layer llm machinelearning natural prior set small softmax thought token tokens training training data weights and biases