[D] Does the existence of mesa optimizers in modern models like transformers make instrumental convergence (think paperclip maximizer) scenarios unlikely?

Recent work shows transformers are capable of performing multi-step gradient descent of mesa objectives inside of their transformer layers. This is even possible for linear transformers, which effectively perform linear optimization on deep representations of features calculated by earlier layers.

[https://arxiv.org/pdf/2309.05858.pdf](https://arxiv.org/pdf/2309.05858.pdf)

For those unfamiliar, [instrumental convergence](https://en.wikipedia.org/wiki/Instrumental_convergence) is the idea that entities with different goals will tend towards different subgoals. Examples could include gathering power, not dying, acquiring resources, etc. A famous thought experiment, known as the paperclip maximizer, is the …

convergence features gradient inside linear machinelearning mesa modern optimization shows think transformer transformers work