LayerAct: Advanced activation mechanism utilizing layer-direction normalization for CNNs with BatchNorm

In this work, we propose a novel activation mechanism aimed at establishing layer-level activation (LayerAct) functions for CNNs with BatchNorm. These functions are designed to be more noise-robust compared to existing element-level activation functions by reducing the layer-level fluctuation of the activation outputs due to shift in inputs. Moreover, the LayerAct functions achieve this noise-robustness independent of the activation's saturation state, which limits the activation output space and complicates efficient training. We present an analysis and experiments demonstrating that LayerAct …

advanced cnns cs.cv cs.lg cs.ne element functions inputs layer noise normalization novel robust shift work