Three Quantization Regimes for ReLU Networks

arXiv:2405.01952v1 Announce Type: cross
Abstract: We establish the fundamental limits in the approximation of Lipschitz functions by deep ReLU neural networks with finite-precision weights. Specifically, three regimes, namely under-, over-, and proper quantization, in terms of minimax approximation error behavior as a function of network weight precision, are identified. This is accomplished by deriving nonasymptotic tight lower and upper bounds on the minimax approximation error. Notably, in the proper-quantization regime, neural networks exhibit memory-optimality in the approximation of Lipschitz functions. …

abstract approximation arxiv behavior cs.ai cs.it cs.lg error function functions fundamental math.it minimax network networks neural networks precision quantization relu stat.ml terms type