Dropout Regularization Versus $\ell_2$-Penalization in the Linear Model

arXiv:2306.10529v2 Announce Type: replace-cross
Abstract: We investigate the statistical behavior of gradient descent iterates with dropout in the linear regression model. In particular, non-asymptotic bounds for the convergence of expectations and covariance matrices of the iterates are derived. The results shed more light on the widely cited connection between dropout and l2-regularization in the linear model. We indicate a more subtle relationship, owing to interactions between the gradient descent dynamics and the additional randomness induced by dropout. Further, we study …

abstract arxiv behavior convergence covariance dropout gradient light linear linear model linear regression math.st regression regularization results statistical stat.ml stat.th type