ScaLearn: Simple and Highly Parameter-Efficient Task Transfer by Learning to Scale

arXiv:2310.01217v3 Announce Type: replace
Abstract: Multi-task learning (MTL) has shown considerable practical benefits, particularly when using language models (LMs). While this is commonly achieved by learning $n$ tasks under a joint optimization procedure, some methods, such as AdapterFusion, divide the problem into two stages: (i) task learning, where knowledge specific to a task is encapsulated within sets of parameters (e.g., adapters), and (ii) transfer, where this already learned knowledge is leveraged for a target task. This separation of concerns provides …

abstract arxiv benefits cs.ai cs.cl cs.lg knowledge language language models lms multi-task learning optimization practical replace scale simple tasks transfer type while