[D] Removed 50% of the weights from a top leaderboard LLM without negatively impacting the evals

I removed 50% of the weights from a top leaderboard LLM without negatively impacting the evals.

Using SparseML I was able to zero out 50% of the

SOLAR-10.7B-Instruct-v1.0 weights.

I then quantized the remaining weights to INT8.

The results are amazing!

​

https://preview.redd.it/uefy5u1hin7c1.png?width=927&format=png&auto=webp&s=35f9c3a07ab3e7f3a0e22a7528adeafc71c4e8e5

Even after pruning and quantizing the model to 50% I still got stellar zero-shot evaluation results.

​

Try the model:

​

https://preview.redd.it/r5tmixshin7c1.png?width=1999&format=png&auto=webp&s=61370090bb0083fecde7b00310bda71527e2eb61

Interestingly, the model is pruned and quantized in one shot. This means that no retraining …

evals leaderboard llm machinelearning solar