[P] llm_microlibs: Building blocks for running models in distributed mode under budget constraints

Link to project: https://github.com/microlib-org/llm_microlibs

Documentation is currently lacking, coming soon

Warning: very long post. TLDR: this post answers some questions I had about generating text with full, unquantized Falcon-180B under budget constraints.

# What is the goal

The goal is to benchmark full, unquantized Falcon-180B. I chose Falcon-180B because it is the biggest open-source model available currently. I also do not use any optimization such as speculative decoding or any kind of quantization. I benchmark both for small and large …

benchmark budget constraints documentation falcon long post machinelearning optimization questions text