[R] Variational Inference: Reverse KL vs. Forward KL

Hi all,

I'm working on variational inference methods, mainly in the context of BNNs. Using the reverse (exclusive) KL as the variational objective is the common approach, though lately I stumbled upon some interesting works that use the forward (inclusive) KL as an objective instead, e.g \[1\]\[2\]\[3\]. Also in the context of VI for GPs both divergence measures have been used, see e.g \[4\].

While I'm familiar with the well-known difference between the objectives that the reverse KL is 'mode-seeking' …

context exclusive inference machinelearning