Towards Robust Multi-Modal Reasoning via Model Selection

arXiv:2310.08446v2 Announce Type: replace
Abstract: The reasoning capabilities of LLM (Large Language Model) are widely acknowledged in recent research, inspiring studies on tool learning and autonomous agents. LLM serves as the "brain" of the agent, orchestrating multiple tools for collaborative multi-step task solving. Unlike methods invoking tools like calculators or weather APIs for straightforward tasks, multi-modal agents excel by integrating diverse AI models for complex challenges. However, current multi-modal agents neglect the significance of model selection: they primarily focus on …

arxiv cs.ai cs.lg modal model selection multi-modal reasoning robust type via