[D] Anyone else notice a surprisingly big difference between FP32 and FP16 models?

At the company I work at we run an image classification service. The model's fairly simple: we just have a pre-trained image encoder backbone model that we further pre-trained with our own data, and we have a fully-connected layer for each "task." A "task" could be something like classifying images of dogs or classifying the color that an image has.

I recently changed my code to use HuggingFace's Accelerate module rather than PyTorch's native DDP and also was training my …

big classification data difference encoder fp16 image layer machinelearning service simple something the company work