Information Bottleneck Analysis of Deep Neural Networks via Lossy Compression

arXiv:2305.08013v2 Announce Type: replace
Abstract: The Information Bottleneck (IB) principle offers an information-theoretic framework for analyzing the training process of deep neural networks (DNNs). Its essence lies in tracking the dynamics of two mutual information (MI) values: between the hidden layer output and the DNN input/target. According to the hypothesis put forth by Shwartz-Ziv & Tishby (2017), the training process consists of two distinct phases: fitting and compression. The latter phase is believed to account for the good generalization performance …

abstract analysis arxiv compression cs.it cs.lg dnn dynamics framework hidden hypothesis information layer lies math.it networks neural networks process the information tracking training type values via