Gradient Sketches for Training Data Attribution and Studying the Loss Landscape

Random projections or sketches of gradients and Hessian vector products play an essential role in applications where one needs to store many such vectors while retaining accurate information about their relative geometry. Two important scenarios are training data attribution (tracing a model's behavior to the training data), where one needs to store a gradient for each training example, and the study of the spectrum of the Hessian (to analyze the training dynamics), where one needs to store multiple Hessian vector …

applications attribution behavior cs.lg data geometry gradient information landscape loss products random role stat.ml store studying tracing training training data vector vectors