Temperature Scaling and Beam Search Text Generation in LLMs, for the ML-Adjacent

What “temperature” is, how it works, its relationship to the beam search heuristic, and how LLM output generation can still go haywire

Photo by Paul Green on Unsplash; all other images by author unless otherwise noted

If you’ve spent any time with APIs for LLMs like those from OpenAI or Anthropic, you’ll have seen the temperature setting available in the API. How is this parameter used, and how does it work?

From the Anthropic chat API documentation: …

algorithms deep-dives llm machine learning naturallanguageprocessing