LLMR: Real-time Prompting of Interactive Worlds using Large Language Models

arXiv:2309.12276v3 Announce Type: replace-cross
Abstract: We present Large Language Model for Mixed Reality (LLMR), a framework for the real-time creation and modification of interactive Mixed Reality experiences using LLMs. LLMR leverages novel strategies to tackle difficult cases where ideal training data is scarce, or where the design goal requires the synthesis of internal dynamics, intuitive analysis, or advanced interactivity. Our framework relies on text interaction and the Unity game engine. By incorporating techniques for scene understanding, task planning, self-debugging, and …

abstract arxiv cases cs.ai cs.cl cs.et cs.hc data design framework interactive language language model language models large language large language model large language models llms mixed mixed reality novel prompting reality real-time strategies training training data type