DeFT: Flash Tree-attention with IO-Awareness for Efficient Tree-search-based LLM Inference

arXiv:2404.00242v1 Announce Type: new
Abstract: Decoding using tree search can greatly enhance the inference quality for transformer-based Large Language Models (LLMs). Depending on the guidance signal, it searches for the best path from root to leaf in the tree by forming LLM outputs to improve controllability, reasoning ability, alignment, et cetera. However, current tree decoding strategies and their inference systems do not suit each other well due to redundancy in computation, memory footprints, and memory access, resulting in inefficient inference. …

abstract alignment arxiv attention cs.ai cs.cl decoding flash guidance inference language language models large language large language models llm llms path quality reasoning search signal transformer tree type