What to Do When Your Discrete Optimization Is the Size of a Neural Network?

arXiv:2402.10339v1 Announce Type: new
Abstract: Oftentimes, machine learning applications using neural networks involve solving discrete optimization problems, such as in pruning, parameter-isolation-based continual learning and training of binary networks. Still, these discrete problems are combinatorial in nature and are also not amenable to gradient-based optimization. Additionally, classical approaches used in discrete settings do not scale well to large neural networks, forcing scientists and empiricists to rely on alternative methods. Among these, two main distinct sources of top-down information can be …

abstract applications arxiv binary continual cs.lg gradient machine machine learning machine learning applications nature network networks neural network neural networks optimization pruning training type