Human Language Modeling. (arXiv:2205.05128v1 [cs.CL])

Web: http://arxiv.org/abs/2205.05128

Natural language is generated by people, yet traditional language modeling
views words or documents as if generated independently. Here, we propose human
language modeling (HuLM), a hierarchical extension to the language modeling
problem whereby a human-level exists to connect sequences of documents (e.g.
social media messages) and capture the notion that human language is moderated
by changing human states. We introduce, HaRT, a large-scale transformer model
for the HuLM task, pre-trained on approximately 100,000 social media users, and
demonstrate its …

arxiv human language modeling