Polynomial Width is Sufficient for Set Representation with High-dimensional Features

arXiv:2307.04001v3 Announce Type: replace
Abstract: Set representation has become ubiquitous in deep learning for modeling the inductive bias of neural networks that are insensitive to the input order. DeepSets is the most widely used neural network architecture for set representation. It involves embedding each set element into a latent space with dimension $L$, followed by a sum pooling to obtain a whole-set embedding, and finally mapping the whole-set embedding to the output. In this work, we investigate the impact of …

abstract architecture arxiv become bias cs.lg deep learning element embedding features inductive modeling network network architecture networks neural network neural networks polynomial representation set space type