[D]Three things I think should get more attention in large language models

1. **Tokenization Techniques**: Many people use the default BPE tokenizer for llama2 or other common tokenizers. But I think we could do a lot of experiments with different kinds of tokenizers, especially ones that are made to work well with certain types of data. The size of the vocabulary is a really important setting when you're working with big language models. You could try using a much smaller vocabulary and tokenizer for a data set that only includes certain words, …

attention data language language models large language large language models llama2 machinelearning people think tokenization types work