Training CausalLM Models Part 1: What Actually Is CausalLM?

The first part of a practical guide to using HuggingFace’s CausalLM class

Causal langauge models model each new word as a function of all previous words. Source: Pexels

If you’ve played around with recent models on HuggingFace, chances are you encountered a causal language model. When you pull up the documentation for a model family, you’ll get a page with “tasks” like LlamaForCausalLM or LlamaForSequenceClassification.

If you’re like me, going from that documentation to actually finetuning a model can …

fine-tuning hugging face large language models machine learning nlp