Calibration of Transformer architectures

What are the most reliable methods to make transformers output a low likelihood on rare/unobserved sequences? In other words, I train my transformer to model a certain type of sequences. I'd like that, given an out-of-distribution sequence or a very rare sequence, the transformer would assign to it a low likelihood.

I guess this would mostly happen without any additional technique for rare sequences, but I'm afraid that out-of-distribution sequences might cause troubles. Do you know any work in likelihood …

architectures deeplearning distribution likelihood low train transformer transformers type words