[D] a sentence level transformer to improve memory for a token level transformer?

I have an (probably dumb) idea for long term transformer memory.

You can embed sentences into vectors of length \~128 - \~2048 right? Then you can cluster those sentences and effectively project them into lower dimensional spaces.

I have often wondered whether you could take \~50.000 cardinal points in the embedding space (points such that the summed of squared distance to all sentences in a representative corpus is minimal). You'd then map each sentence in a big corpus to the …

cluster embed machinelearning memory project spaces them token transformer vectors