PopulAtion Parameter Averaging (PAPA)

arXiv:2304.03094v3 Announce Type: replace
Abstract: Ensemble methods combine the predictions of multiple models to improve performance, but they require significantly higher computation costs at inference time. To avoid these costs, multiple neural networks can be combined into one by averaging their weights. However, this usually performs significantly worse than ensembling. Weight averaging is only beneficial when different enough to benefit from combining them, but similar enough to average well. Based on this idea, we propose PopulAtion Parameter Averaging (PAPA): a …

abstract arxiv computation costs cs.cv cs.lg ensemble however inference multiple networks neural networks performance population predictions type