Beyond automatic differentiation

Posted by Matthew Streeter, Software Engineer, Google Research

Derivatives play a central role in optimization and machine learning. By locally approximating a training loss, derivatives guide an optimizer toward lower values of the loss. Automatic differentiation frameworks such as TensorFlow, PyTorch, and JAX are an essential part of modern machine learning, making it feasible to use gradient-based optimizers to train very complex models.

But are derivatives all we need? By themselves, derivatives only tell us how a …

algorithms beyond derivatives differentiation engineer example frameworks function google google research gradient guide jax loss machine machine learning making optimization part pytorch research role scale software software engineer tensorflow training training loss values