Glyph-ByT5: A Customized Text Encoder for Accurate Visual Text Rendering

arXiv:2403.09622v1 Announce Type: new
Abstract: Visual text rendering poses a fundamental challenge for contemporary text-to-image generation models, with the core problem lying in text encoder deficiencies. To achieve accurate text rendering, we identify two crucial requirements for text encoders: character awareness and alignment with glyphs. Our solution involves crafting a series of customized text encoder, Glyph-ByT5, by fine-tuning the character-aware ByT5 encoder using a meticulously curated paired glyph-text dataset. We present an effective method for integrating Glyph-ByT5 with SDXL, resulting …

abstract alignment arxiv challenge core cs.cv encoder identify image image generation image generation models rendering requirements solution text text-to-image type visual