Metric-Learning Encoding Models Identify Processing Profiles of Linguistic Features in BERT's Representations

arXiv:2402.11608v1 Announce Type: new
Abstract: We introduce Metric-Learning Encoding Models (MLEMs) as a new approach to understand how neural systems represent the theoretical features of the objects they process. As a proof-of-concept, we apply MLEMs to neural representations extracted from BERT, and track a wide variety of linguistic features (e.g., tense, subject person, clause type, clause embedding). We find that: (1) linguistic features are ordered: they separate representations of sentences to different degrees in different layers; (2) neural representations are …

abstract apply arxiv bert concept cs.cl encoding features identify metric-learning objects process processing profiles proof-of-concept systems type