The Pyramid of Captions

arXiv:2405.00485v1 Announce Type: new
Abstract: We introduce a formal information-theoretic framework for image captioning by regarding it as a representation learning task. Our framework defines three key objectives: task sufficiency, minimal redundancy, and human interpretability. Building upon this foundation, we propose a novel Pyramid of Captions (PoCa) method, which constructs caption pyramids by generating localized captions for zoomed-in image patches and integrating them with global caption information using large language models. This approach leverages intuition that the detailed examination of …

abstract arxiv building captioning captions cs.cv foundation framework human image information interpretability key novel pyramid redundancy representation representation learning type