Oct. 31, 2022, 3:10 a.m. | /u/akardashian

Natural Language Processing www.reddit.com

I am sorry if this is a dumb question!

In short, I am trying to initialize a model from a pre-trained encoder and its LM head. For BERT, I can use BertModel and grab the LM head from BERTForPretraining. I am trying to do the same for RoBERTa, but there is no equivalent RoBERTaForPretraining from HF. I am thinking about grabbing the LM head from RoBERTaForMaskedLM instead, but I am not sure if it would be compatible with the RoBERTa …

huggingface languagetechnology roberta

AI Research Scientist

@ Vara | Berlin, Germany and Remote

Data Architect

@ University of Texas at Austin | Austin, TX

Data ETL Engineer

@ University of Texas at Austin | Austin, TX

Lead GNSS Data Scientist

@ Lurra Systems | Melbourne

Senior Data Engineer (m/f/d)

@ Project A Ventures | Berlin, Germany

Principle Research Scientist

@ Analog Devices | US, MA, Boston