Explicitly Representing Syntax Improves Sentence-to-layout Prediction of Unexpected Situations

arXiv:2401.14212v2 Announce Type: replace
Abstract: Recognizing visual entities in a natural language sentence and arranging them in a 2D spatial layout require a compositional understanding of language and space. This task of layout prediction is valuable in text-to-image synthesis as it allows localized and controlled in-painting of the image. In this comparative study it is shown that we can predict layouts from language representations that implicitly or explicitly encode sentence syntax, if the sentences mention similar entity-relationships to the ones …

abstract arxiv cs.cl image language natural natural language painting prediction space spatial syntax synthesis text text-to-image them type understanding visual