MTGA: Multi-view Temporal Granularity aligned Aggregation for Event-based Lip-reading

arXiv:2404.11979v1 Announce Type: new
Abstract: Lip-reading is to utilize the visual information of the speaker's lip movements to recognize words and sentences. Existing event-based lip-reading solutions integrate different frame rate branches to learn spatio-temporal features of varying granularities. However, aggregating events into event frames inevitably leads to the loss of fine-grained temporal information within frames. To remedy this drawback, we propose a novel framework termed Multi-view Temporal Granularity aligned Aggregation (MTGA). Specifically, we first present a novel event representation method, …

abstract aggregation arxiv cs.cv event events features fine-grained however information leads learn loss movements rate reading solutions speaker temporal type view visual words