Comparing Pre-trained Human Language Models: Is it Better with Human Context as Groups, Individual Traits, or Both?

arXiv:2401.12492v2 Announce Type: replace-cross
Abstract: Incorporating human context into language models is the next frontier for human-centered natural language processing. Currently, two pre-training methods exist: group-wise attributes (e.g., over-45-year-olds) or individual traits. Group attributes are coarse -- not all 45-year-olds write the same way -- while modeling individual traits allows for a more personalized representation, but requires more complex modeling and data. So far, it is unclear which pre-training approach benefits what tasks. We compare pre-training models with human context …

abstract arxiv context cs.ai cs.cl cs.lg human language language models language processing natural natural language natural language processing next pre-training processing training type wise