[D] A look at Attention and Transformers from the lens of Inductive Bias in ML

Hello people, posting a video from my YT channel about Transformers and the interesting paradigm shift they brought into deep learning research. Before attention, the trend used to be about adding more inductive bias into the model architecture (CNNs with their location bias, RNNs with their temporal bias)… The success of transformers kinda showed that general architectures can trump inductive bias when you got enough data and a large-ass model to train.

architecture attention bias cnns deep learning hello inductive location look machinelearning paradigm people research shift transformers trend video