[D] What are the major general advances in techniques?

Hey, I'm a casual observer of the DL space, what are the biggest technique changes or discoveries that are now used everywhere? From my view:

- ReLU - simple to train non-linear function
- Dropout - how to not overfit (2014)
- Residual connections - how to go deep (2015)
- Layer normalisation - how to fit better (2016)
- Transformers - how to train sequences in parallel (2017)

What's the other improvements or discoveries? More general the idea the …

general machinelearning major