Questions about BigBird

Hello, people. I still have some questions after reading the paper about Big Bird model ( https://arxiv.org/pdf/2007.14062v2.pdf ) and will be happy if some Big Bird specialists will help me to understand this model better.

1. Is distribution of random attention (Figure 1 (a)) fixed from advance for all inputs, or it somehow can be different for different inputs even on the same head?
2. In BIGBIRD-ETC, do they add some additional global tokens, aside of [CLS]?
3. In BIGBIRD-ITC, how is the …

languagetechnology