Leveraging Continuously Differentiable Activation Functions for Learning in Quantized Noisy Environments

Real-world analog systems intrinsically suffer from noise that can impede model convergence and accuracy on a variety of deep learning models. We demonstrate that differentiable activations like GELU and SiLU enable robust propagation of gradients which help to mitigate analog quantization error that is ubiquitous to all analog systems. We perform analysis and training of convolutional, linear, and transformer networks in the presence of quantized noise. Here, we are able to demonstrate that continuously differentiable activation functions are significantly more …

accuracy analog convergence cs.lg deep learning differentiable environments error functions noise propagation quantization robust systems world