The Simpler The Better: An Entropy-Based Importance Metric To Reduce Neural Networks' Depth

arXiv:2404.18949v1 Announce Type: new
Abstract: While deep neural networks are highly effective at solving complex tasks, large pre-trained models are commonly employed even to solve consistently simpler downstream tasks, which do not necessarily require a large model's complexity. Motivated by the awareness of the ever-growing AI environmental impact, we propose an efficiency strategy that leverages prior knowledge transferred by large models. Simple but effective, we propose a method relying on an Entropy-bASed Importance mEtRic (EASIER) to reduce the depth of …

abstract arxiv complexity cs.lg entropy environmental ever importance networks neural networks pre-trained models reduce solve tasks type while