Breaking the HISCO Barrier: Automatic Occupational Standardization with OccCANINE

arXiv:2402.13604v1 Announce Type: new
Abstract: This paper introduces a new tool, OccCANINE, to automatically transform occupational descriptions into the HISCO classification system. The manual work involved in processing and classifying occupational descriptions is error-prone, tedious, and time-consuming. We finetune a preexisting language model (CANINE) to do this automatically thereby performing in seconds and minutes what previously took days and weeks. The model is trained on 14 million pairs of occupational descriptions and HISCO codes in 13 different languages contributed by …

abstract arxiv breaking classification cs.cl econ.em error language language model paper processing standardization tool type work