March 10, 2024, 6:10 p.m. | /u/datashri

Machine Learning www.reddit.com

I'm starting to get a hang.of the attention paper and the significance of Q, K and V and of dot product attention and multi head attention.

What i don't understand is how the values of Q, K, and V matrices are trained. I've read the cross validation stack exchange answers and some others. I get that QKV come from the previous layer, but how does the previous layer determine/train their values?

Some form of backprop, sure. But what's the goal …

attention head machinelearning paper product significance stack stack exchange training validation values

Founding AI Engineer, Agents

@ Occam AI | New York

AI Engineer Intern, Agents

@ Occam AI | US

AI Research Scientist

@ Vara | Berlin, Germany and Remote

Data Architect

@ University of Texas at Austin | Austin, TX

Data ETL Engineer

@ University of Texas at Austin | Austin, TX

Lead GNSS Data Scientist

@ Lurra Systems | Melbourne