How to Fit Large Language Models in Small Memory: Quantization

Large Language Models can be used for text generation, translation, question-answering tasks, etc. However, LLMs are also very large (obviously, Large language models) and require a lot of memory. This can make them challenging for small devices like phones and tablets.

Multiply the parameters by the chosen precision size to determine the model size in bytes. Let’s say the precision we’ve chosen is float16 (16 bits = 2 bytes). Let’s say we want to use the BLOOM-176B model. We need …

devices etc langchain language language models large language large language models llm llms memory phones precision quantization small tablets tasks text text generation them translation