Neural Networks with A La Carte Selection of Activation Functions. (arXiv:2206.12166v1 [cs.NE])

Activation functions (AFs), which are pivotal to the success (or failure) of
a neural network, have received increased attention in recent years, with
researchers seeking to design novel AFs that improve some aspect of network
performance. In this paper we take another direction, wherein we combine a slew
of known AFs into successful architectures, proposing three methods to do so
beneficially: 1) generate AF architectures at random, 2) use Optuna, an
automatic hyper-parameter optimization software framework, with a
Tree-structured Parzen …

arxiv la networks neural networks