Q: predicting only one token in autoregressive model training?

I'm training an autoregressive model (transformer encoder + decoder)

where a text is given as input and output is also text(both tokenized).

I've been using nll loss, and even though the nll loss decreased significantly, the predictions are just a bunch of repetition of a single token.

for example,

input: hello world

output: aaaaaaaaaaaaaaaaa

​

I've been looking in to the model architecture and loss function, but can't catch any bugs in it yet.

​

Could you suggest any tips …

autoregressive model deeplearning training