[D] benefits of using only attention weights for LoRA

I'm confused as to why people would only use a subset of weights as learnable parameters for LoRA. If you are only using attention weights as update params, you still need the decomposed weights for the other layers to get the derivative of the loss with respect to the attention weights. That's how the chain rule works, so I don't see how it would help with memory consumption. Is there something I'm missing here?

attention benefits lora loss machinelearning parameters params people