T3: Transparent Tracking & Triggering for Fine-grained Overlap of Compute & Collectives. (arXiv:2401.16677v1 [cs.AR])

Large Language Models increasingly rely on distributed techniques for their
training and inference. These techniques require communication across devices
which can reduce scaling efficiency as the number of devices increases. While
some distributed techniques can overlap, and thus, hide this communication with
independent computations, techniques such as Tensor Parallelism (TP) inherently
serialize communication with model execution. One approach to hide this
serialized communication is to interleave it with the producer operation (of
the communicated data) in a fine-grained manner. However, …

arxiv communication compute cs.ar devices distributed efficiency fine-grained hide independent inference language language models large language large language models reduce scaling tracking training