March 9, 2022, 7:54 p.m. | Chaim Rand

Towards Data Science - Medium towardsdatascience.com

How to debug occurrences of a NaN loss in your training project

Photo by Alan Emery on Unsplash

In a previous post, we attempted to offer some support in the — often difficult, sometimes impossible, and always maddening — task of debugging in TensorFlow. The blog includes a description of, what I believe to be, the ultimate example of the potential suffering of the modern day machine learning developer — the sudden appearance of the NaN loss. In the …

artificial intelligence debugging deep learning nan tensorflow training

Founding AI Engineer, Agents

@ Occam AI | New York

AI Engineer Intern, Agents

@ Occam AI | US

AI Research Scientist

@ Vara | Berlin, Germany and Remote

Data Architect

@ University of Texas at Austin | Austin, TX

Data ETL Engineer

@ University of Texas at Austin | Austin, TX

Lead GNSS Data Scientist

@ Lurra Systems | Melbourne