On Parameter Efficiency of Neural Language Models

Abstract: Pre-trained neural language models have demonstrated remarkable generalizability in various downstream tasks, such as natural language understanding and question answering. However, these models have grown to contain hundreds of billions of parameters, making them difficult to be deployed in applications with latency requirements and memory constraints. Furthermore, existing research have demonstrated the existence of significant redundant parameters in neural language models. Such redundancy can further compromise their downstream generalizability. To tackle these challenges, my research focus on training neural …

abstract applications constraints efficiency language language models language understanding latency making memory natural natural language parameters question answering requirements research tasks them understanding