Splitting GPT-J And Other Large Language Models Over Multiple GPUs

As models get larger and larger, it becomes less feasible to load a model onto a single GPU. This video goes over using Parallelformers, a library that allows one to easily split GPT-J and other large models over multiple GPUs. This allows one with multiple 3060s 3080s 3080Tis etc to easily use this large model with CUDA acceleration when previously it was very difficult.

https://www.youtube.com/watch?v=CxCxXI2m2a0

deeplearning gpt-j language language models large language models