Representing Online Handwriting for Recognition in Large Vision-Language Models

arXiv:2402.15307v1 Announce Type: cross
Abstract: The adoption of tablets with touchscreens and styluses is increasing, and a key feature is converting handwriting to text, enabling search, indexing, and AI assistance. Meanwhile, vision-language models (VLMs) are now the go-to solution for image understanding, thanks to both their state-of-the-art performance across a variety of tasks and the simplicity of a unified approach to training, fine-tuning, and inference. While VLMs obtain high performance on image-based tasks, they perform poorly on handwriting recognition when …

abstract adoption ai assistance art arxiv cs.ai cs.cv cs.lg enabling feature handwriting image indexing key language language models performance recognition search solution state tablets text type understanding vision vision-language models vlms