PALBERT: Teaching ALBERT to Ponder. (arXiv:2204.03276v1 [cs.LG])

Currently, pre-trained models can be considered the default choice for a wide
range of NLP tasks. Despite their SoTA results, there is practical evidence
that these models may require a different number of computing layers for
different input sequences, since evaluating all layers leads to overconfidence
on wrong predictions (namely overthinking). This problem can potentially be
solved by implementing adaptive computation time approaches, which were first
designed to improve inference speed. Recently proposed PonderNet may be a
promising solution for …

arxiv teaching