FoodLMM: A Versatile Food Assistant using Large Multi-modal Model

arXiv:2312.14991v2 Announce Type: replace
Abstract: Large Multi-modal Models (LMMs) have made impressive progress in many vision-language tasks. Nevertheless, the performance of general LMMs in specific domains is still far from satisfactory. This paper proposes FoodLMM, a versatile food assistant based on LMMs with various capabilities, including food recognition, ingredient recognition, recipe generation, nutrition estimation, food segmentation and multi-round conversation. To facilitate FoodLMM to deal with tasks beyond pure text output, we introduce a series of novel task-specific tokens and heads, …

abstract arxiv assistant capabilities cs.cv domains food general language lmms modal multi-modal paper performance progress recipe recognition tasks type vision