What do Transformers Know about Government?

arXiv:2404.14270v1 Announce Type: cross
Abstract: This paper investigates what insights about linguistic features and what knowledge about the structure of natural language can be obtained from the encodings in transformer language models.In particular, we explore how BERT encodes the government relation between constituents in a sentence. We use several probing classifiers, and data from two morphologically rich languages. Our experiments show that information about government is encoded across all transformer layers, but predominantly in the early layers of the model. …

abstract arxiv bert classifiers cs.cl cs.lg data explore features government insights knowledge language language models natural natural language paper transformer transformer language models transformers type