Reducing Model Jitter: Stable Re-training of Semantic Parsers in Production Environments. (arXiv:2204.04735v2 [cs.CL] UPDATED)

Retraining modern deep learning systems can lead to variations in model
performance even when trained using the same data and hyper-parameters by
simply using different random seeds. We call this phenomenon model jitter. This
issue is often exacerbated in production settings, where models are retrained
on noisy data. In this work we tackle the problem of stable retraining with a
focus on conversational semantic parsers. We first quantify the model jitter
problem by introducing the model agreement metric and showing …

arxiv environments production production environments semantic training