Hitting "Probe"rty with Non-Linearity, and More

arXiv:2402.16168v1 Announce Type: new
Abstract: Structural probes learn a linear transformation to find how dependency trees are embedded in the hidden states of language models. This simple design may not allow for full exploitation of the structure of the encoded information. Hence, to investigate the structure of the encoded information to its full extent, we incorporate non-linear structural probes. We reformulate the design of non-linear structural probes introduced by White et al. making its design simpler yet effective. We also …

abstract arxiv cs.ai cs.cl design embedded exploitation hidden information language language models learn linear probe simple transformation trees type