Three Decades of Activations: A Comprehensive Survey of 400 Activation Functions for Neural Networks

arXiv:2402.09092v1 Announce Type: new
Abstract: Neural networks have proven to be a highly effective tool for solving complex problems in many areas of life. Recently, their importance and practical usability have further been reinforced with the advent of deep learning. One of the important conditions for the success of neural networks is the choice of an appropriate activation function introducing non-linearity into the model. Many types of these functions have been proposed in the literature in the past, but there …

abstract arxiv cs.lg cs.ne deep learning functions importance life networks neural networks practical survey tool type usability