[D] How do byte-level language models work?

I've recently been trying to pre-train my own small language model on the tiny-series datasets on huggingface. I also wanted to use a model similar to MEGABYTE but I don't understand how using bytes would work. The only implementation I could find from lucidrains used str(chr(max(32, token))) to decode any token (byte) to a character and put the embedding size as 256. Firstly, why 256 and not 256-32 as any values below 32 are ignored? Also, many byte-level models including …

datasets decode huggingface implementation language language model language models machinelearning max series small token work