Competing Mutual Information Constraints with Stochastic Competition-based Activations for Learning Diversified Representations. (arXiv:2201.03624v1 [cs.LG])

This work aims to address the long-established problem of learning
diversified representations. To this end, we combine information-theoretic
arguments with stochastic competition-based activations, namely Stochastic
Local Winner-Takes-All (LWTA) units. In this context, we ditch the conventional
deep architectures commonly used in Representation Learning, that rely on
non-linear activations; instead, we replace them with sets of locally and
stochastically competing linear units. In this setting, each network layer
yields sparse outputs, determined by the outcome of the competition between
units that …

arxiv competition information learning stochastic