[D] Are shared weights in transformer architecture also receiving the same gradient updates?

Hi all,

I have built a transformer model in Pytorch which works really well. I created shared weights in the source and target embedding matrices as well as the final logits layer before the Softmax function. I was reading online and also section 3 of the 'attention is all you need' paper.

Pre-training the weights in these specific matrices are identical. However, after training, they are also identical (but the model predicts perfectly etc, losses have decreased and it has …

architecture attention embedding function gradient machinelearning pytorch reading softmax transformer transformer architecture transformer model updates