Prompt Lookup Decoding

Prompt Lookup Decoding

Really neat LLM optimization trick by Apoorv Saxena, who observed that it's common for sequences of tokens in LLM input to be reflected by the output - snippets included in a summarization, for example.

Apoorv's code performs a simple search for such prefixes and uses them to populate a set of suggested candidate IDs during LLM token generation.

The result appears to provide around a 2.4x speed-up in generating outputs!

Via @abacaj

ai code decoding example generativeai llm llms optimization prompt search set simple summarization them tokens trick