Nonsmooth Implicit Differentiation: Deterministic and Stochastic Convergence Rates

arXiv:2403.11687v1 Announce Type: cross
Abstract: We study the problem of efficiently computing the derivative of the fixed-point of a parametric non-differentiable contraction map. This problem has wide applications in machine learning, including hyperparameter optimization, meta-learning and data poisoning attacks. We analyze two popular approaches: iterative differentiation (ITD) and approximate implicit differentiation (AID). A key challenge behind the nonsmooth setting is that the chain rule does not hold anymore. Building upon the recent work by Bolte et al. (2022), who proved …

abstract analyze applications arxiv attacks computing convergence cs.lg data data poisoning differentiable differentiation fixed-point hyperparameter iterative machine machine learning map math.oc meta meta-learning optimization parametric poisoning attacks popular stat.ml stochastic study type