Improving Covariance Conditioning of the SVD Meta-layer by Orthogonality. (arXiv:2207.02119v1 [cs.CV])

Inserting an SVD meta-layer into neural networks is prone to make the
covariance ill-conditioned, which could harm the model in the training
stability and generalization abilities. In this paper, we systematically study
how to improve the covariance conditioning by enforcing orthogonality to the
Pre-SVD layer. Existing orthogonal treatments on the weights are first
investigated. However, these techniques can improve the conditioning but would
hurt the performance. To avoid such a side effect, we propose the Nearest
Orthogonal Gradient (NOG) and …

arxiv covariance cv meta svd