Tutorial to improve GPT throughput 16 times with dynamic batching

I wrote a tutorial to improve GPT completion throughput with dynamic batching [https://microsoft.github.io/batch-inference/examples/gpt\_completion.html](https://microsoft.github.io/batch-inference/examples/gpt_completion.html). And I can achieve 16 times throughput on V100 comparing to baseline. We built a python dynamic batching library so you can apply it on your own models easily [https://github.com/microsoft/batch-inference](https://github.com/microsoft/batch-inference).


Although the tutorial we built for GPT shows promising result on throughput, it doesn't use complex decoding algorithms like top-p or beam search, and we are aware of more advanced batching algorithms for GPT completion. So we're …

advanced algorithms batching building decoding deeplearning future gpt inference library production search shows tutorial