Zero Mean Leaky ReLu

Hi,

At the risk of groans of "not another ReLu activation function variant", I thought I'd share a simple trick to make the (Leaky)ReLu better behaved, in particular to address criticism about the (Leaky)ReLu not being zero-centred.

The simple trick is to offset the (Leaky)ReLu unit by the expectation of the output under a zero-mean normally distributed input:

Zero Mean Leaky ReLu:

y(x) = max(x, a\*x) - k

k=((1 - a)\*s)/sqrt(2\*pi)

y' = a, for y<-k, 1 otherwise

The resulting …

deeplearning function mean normally relu risk simple thought trick