GELU : Gaussian Error Linear Unit Code (Python, TF, Torch)

GELU : Gaussian Error Linear Unit Code (Python, TF, Torch)

Code tutorial for GELU, Gaussian Error Linear Unit activation function. Includes bare python, Tensorflow and Pytorch code.

GELU Activation Function

Gaussian Error Linear Unit, GELU, is the most-used activation function in state-of-the-art models including BERT, GPT, Vision Transformers, etc..

If you want to understand the intuition and math behind GELU I suggest you check my previous article covering the GELU paper (GELU, the ReLU …

activation-functions code deep learning error linear neural networks python