Towards Theoretical Analysis of Transformation Complexity of ReLU DNNs. (arXiv:2205.01940v1 [cs.LG])

Web: http://arxiv.org/abs/2205.01940

This paper aims to theoretically analyze the complexity of feature
transformations encoded in DNNs with ReLU layers. We propose metrics to measure
three types of complexities of transformations based on the information theory.
We further discover and prove the strong correlation between the complexity and
the disentanglement of transformations. Based on the proposed metrics, we
analyze two typical phenomena of the change of the transformation complexity
during the training process, and explore the ceiling of a DNN's complexity. The
proposed …

analysis arxiv complexity relu transformation