Generative AI Beyond LLMs: System Implications of Multi-Modal Generation

arXiv:2312.14385v2 Announce Type: replace-cross
Abstract: As the development of large-scale Generative AI models evolve beyond text (1D) generation to include image (2D) and video (3D) generation, processing spatial and temporal information presents unique challenges to quality, performance, and efficiency. We present the first work towards understanding this new system design space for multi-modal text-to-image (TTI) and text-to-video (TTV) generation models. Current model architecture designs are bifurcated into 2 categories: Diffusion- and Transformer-based models. Our systematic performance characterization on a suite …

abstract ai models arxiv beyond challenges cs.dc cs.lg cs.mm design development efficiency generative generative ai models image information llms modal multi-modal performance processing quality scale space spatial temporal text type understanding unique video work