[D] Where do Lora memory gains come from ? (apart from the optimizer state)

Hello,

I've been reading explanations of Lora for hours now, and there is something I can't wrap my head around: the memory gains. I understand that a lot is gained with the optimizer state that are not needed for frozen layers.

However, in the Lora Paper (Chapter 4.2) it is stated that

>We also observe a 25% speedup during training on GPT-3 175B compared to full fine-tuning5 as we do not need to calculate the gradient for the vast majority …

head hello however lora machinelearning memory paper reading something state