[D] How Exactly does Fuyu's image to embedding with nn.Linear work? Could you do more with it?

As I was asking above, I've been looking at the Fuyu 8b model, and I've been able to break it down to

- model takes in text the regular way, text -> tokens -> embeddings
- it also takes image -> embeddings
- it has a vanilla decoder, so only text comes out, they add special tokens around images, so i'm assuming the decoder ignores output images

So, from what I know, nn.Linear takes in a tensor and makes embeddings …

break it down embedding embeddings image linear machinelearning text tokens work