Distillation of BERT-like models: the code

This article is part 2 of a two-part series on distilling BERT-like models in the fashion of DistilBERT. For part one, you may follow this link. If however, you consider that you have a good grasp on DistilBERT’s distillation method, feel free to skip the read.

Much like chemical distillation, we’re going to extract from our model what matters: knowledge. Photo by Elevate on Unsplash

Recap

In case you haven’t noticed, machine learning models have been getting larger and …

bert code data science distillation machine learning nlp transfer learning