Transformer Encoder for Social Science. (arXiv:2208.08005v1 [cs.CL])

High-quality text data has become an important data source for social
scientists. We have witnessed the success of pretrained deep neural network
models, such as BERT and RoBERTa, in recent social science research. In this
paper, we propose a compact pretrained deep neural network, Transformer Encoder
for Social Science (TESS), explicitly designed to tackle text processing tasks
in social science research. Using two validation tests, we demonstrate that
TESS outperforms BERT and RoBERTa by 16.7% on average when the number …

arxiv encoder science social social science transformer