Simplicial Embeddings in Self-Supervised Learning and Downstream Classification. (arXiv:2204.00616v1 [cs.LG])

We introduce Simplicial Embeddings (SEMs) as a way to constrain the encoded
representations of a self-supervised model to $L$ simplices of $V$ dimensions
each using a Softmax operation. This procedure imposes a structure on the
representations that reduce their expressivity for training downstream
classifiers, which helps them generalize better. Specifically, we show that the
temperature $\tau$ of the Softmax operation controls for the SEM
representation's expressivity, allowing us to derive a tighter downstream
classifier generalization bound than that for classifiers …

arxiv classification learning self-supervised learning supervised learning