Order-Based Pre-training Strategies for Procedural Text Understanding

arXiv:2404.04676v1 Announce Type: new
Abstract: In this paper, we propose sequence-based pretraining methods to enhance procedural understanding in natural language processing. Procedural text, containing sequential instructions to accomplish a task, is difficult to understand due to the changing attributes of entities in the context. We focus on recipes, which are commonly represented as ordered instructions, and use this order as a supervision signal. Our work is one of the first to compare several 'order as-supervision' transformer pre-training methods, including Permutation …

abstract arxiv context cs.cl focus language language processing natural natural language natural language processing paper pre-training pretraining processing recipes strategies text text understanding training type understanding