is there any novelty on architecture design/operators level during the last year?

I was looking into recent SOTA models both for NLP and computer vision and seems it is mainly the same concept - attention with slightly different training approach / different number of same layers combined together. Is there any innovation on lower level happening? Ie any new promising architectures, design approaches or operators which were introduced lately?

Thank you.

architecture attention computer computer vision concept deeplearning design innovation nlp operators sota together training vision