On the Optimal Expressive Power of ReLU DNNs and Its Application in Approximation with Kolmogorov Superposition Theorem. (arXiv:2308.05509v1 [cs.LG])

This paper is devoted to studying the optimal expressive power of ReLU deep
neural networks (DNNs) and its application in approximation via the Kolmogorov
Superposition Theorem. We first constructively prove that any continuous
piecewise linear functions on $[0,1]$, comprising $O(N^2L)$ segments, can be
represented by ReLU DNNs with $L$ hidden layers and $N$ neurons per layer.
Subsequently, we demonstrate that this construction is optimal regarding the
parameter count of the DNNs, achieved through investigating the shattering
capacity of ReLU DNNs. …

application approximation arxiv continuous functions linear networks neural networks paper power relu studying superposition theorem