[D] Speed Up in FP32 vs FP16

- Task: Training and Fine Tuning on Single node 2 GPUs
- Model: CLIP ViT-B-32
- Dataset: MSCOCO Captions
- Number of Workers: 4
- Batch Size: 240 in case of FP16 and 160 in case of FP32

For both FP32 and FP16, each epoch is taking around 6 mins.

One of reason I consider is that *majority of time might constitute of data movement* rather GPU processing, as in case of FP32 there's hardly a moment when GPU utilization …

captions case clip dataset fp16 gpus machinelearning node reason speed training vit workers