AdaCat: Adaptive Categorical Discretization for Autoregressive Models. (arXiv:2208.02246v1 [cs.LG])

Autoregressive generative models can estimate complex continuous data
distributions, like trajectory rollouts in an RL environment, image
intensities, and audio. Most state-of-the-art models discretize continuous data
into several bins and use categorical distributions over the bins to
approximate the continuous data distribution. The advantage is that the
categorical distribution can easily express multiple modes and are
straightforward to optimize. However, such approximation cannot express sharp
changes in density without using significantly more bins, making it parameter
inefficient. We propose an …

arxiv autoregressive models lg