Improved weight initialization for deep and narrow feedforward neural network

arXiv:2311.03733v2 Announce Type: replace
Abstract: Appropriate weight initialization settings, along with the ReLU activation function, have become cornerstones of modern deep learning, enabling the training and deployment of highly effective and efficient neural network models across diverse areas of artificial intelligence. The problem of \textquotedblleft dying ReLU," where ReLU neurons become inactive and yield zero output, presents a significant challenge in the training of deep neural networks with ReLU activation function. Theoretical research and various methods have been introduced to …

abstract artificial artificial intelligence arxiv become cs.lg cs.ne deep learning deployment diverse enabling function intelligence modern narrow network neural network neurons relu training type