Who breaks early, looses: goal oriented training of deep neural networks based on port Hamiltonian dynamics. (arXiv:2304.07070v1 [cs.LG])

The highly structured energy landscape of the loss as a function of
parameters for deep neural networks makes it necessary to use sophisticated
optimization strategies in order to discover (local) minima that guarantee
reasonable performance. Overcoming less suitable local minima is an important
prerequisite and often momentum methods are employed to achieve this. As in
other non local optimization procedures, this however creates the necessity to
balance between exploration and exploitation. In this work, we suggest an event
based control …

arxiv control dynamics energy event exploitation exploration function landscape loss networks neural networks optimization performance strategies training work