Please correct my understanding of "memory" in LLMs

I'm trying to understand how GPTs/LLMs work, on a conceptual level and using the correct terminology.

Here's my understanding so far (please correct if I'm wrong):

1. GPTs are **pre-trained** so that for any given input it spits out the statistically best matching output based on its training.
2. It does this token by token, without "understanding" the output, just that this token is often followed by this other token.
3. It gains this knowledge during its training, when the …

artificial gpts llms memory terminology training understanding work