[D] How do transformers memorize facts after a single gradient update?

I guess I would first like a citation of that fact, or for somebody to tell me I made it up. But folk knowledge is that a transformer trained for a single epoch can recall facts that only appear a single time in the training dataset. This implies that a single update is enough to modify the weights to produce the correct output (without catastrophically forgetting other facts).

This is really surprising to me. I would think a single update …

dataset facts gradient knowledge machinelearning recall training transformer transformers update