[R] Preventing mode collapse and overfitting in Seq2Seq transformers

I have a Seq2Seq dataset where a given target sentence can occur for multiple source sentences.

Eg:

S1 -> T1
S2 -> T1
S3 -> T1

I tried to use `BartForConditionalGeneration` and `T5` models as `Seq2Seq` models.

But I notice that after a single epoch the model starts collapsing to produce the same sentence as output regardless of input. And this most likely happens because of the repetition. If I simply eleiminate all repetition in my dataset by picking only …

machinelearning overfitting seq2seq transformers