Stanford AI Researchers Propose ‘LinkBERT’: A New Pretraining Method That Improves Language Model Training with Document Links

👉 LinkBERT consists of three steps:

(1) obtaining links between documents to build a document graph from the text corpus,

(2) creating link-aware training instances from the graph by placing linked documents together, and finally

(3) pretraining the LM with link-aware self-supervised tasks: masked language modeling (MLM) and document relation prediction (DRP).

👉 LinkBERT is especially effective for multi-hop reasoning and few-shot QA (+5% absolute improvement on HotpotQA and TriviaQA)

[*Continue reading*](https://www.marktechpost.com/2022/06/08/stanford-ai-researchers-propose-linkbert-a-new-pretraining-method-that-improves-language-model-training-with-document-links/) *| Check out the*[ ](https://www.prnewswire.com/news-releases/introducing-the-digital-upcycling-project-by-tilda-the-first-ai-artist-by-lg-ai-research-301561017.html)[*paper*](https://arxiv.org/pdf/2203.15827.pdf)*,* [*github*](https://github.com/michiyasunaga/LinkBERT) *and* [*blog …

ai language language model languagetechnology researchers stanford training