Neural Networks with (Low-Precision) Polynomial Approximations: New Insights and Techniques for Accuracy Improvement

arXiv:2402.11224v1 Announce Type: new
Abstract: Replacing non-polynomial functions (e.g., non-linear activation functions such as ReLU) in a neural network with their polynomial approximations is a standard practice in privacy-preserving machine learning. The resulting neural network, called polynomial approximation of neural network (PANN) in this paper, is compatible with advanced cryptosystems to enable privacy-preserving model inference. Using ``highly precise'' approximation, state-of-the-art PANN offers similar inference accuracy as the underlying backbone model. However, little is known about the effect of approximation, and …

abstract accuracy approximation arxiv cs.cr cs.lg functions improvement insights linear low machine machine learning network networks neural network neural networks non-linear paper polynomial practice precision privacy relu standard type