ALTO: An Efficient Network Orchestrator for Compound AI Systems

arXiv:2403.04311v1 Announce Type: cross
Abstract: We present ALTO, a network orchestrator for efficiently serving compound AI systems such as pipelines of language models. ALTO achieves high throughput and low latency by taking advantage of an optimization opportunity specific to generative language models: streaming intermediate outputs. As language models produce outputs token by token, ALTO exposes opportunities to stream intermediate outputs between stages when possible. We highlight two new challenges of correctness and load balancing which emerge when streaming intermediate data …

abstract ai systems arxiv cs.ai cs.cl cs.dc cs.ir generative intermediate language language models latency low low latency network optimization orchestrator pipelines streaming systems token type