[R] Multimodal patch embeddings - a new ViT model

This research project is an exploration of the idea - `what if you could compare a text embedding to each patch embedding of a ViT?`. I tried out a few things and got some promising results.

There are two key ideas here:
1. To have the image embedding as a convex sum of points on a hypersphere, where each point is a patch embedding. This needs a change in the ViT architecture which is explained [here](https://www.tinyvolt.com/research/classless-vit).
2. Constrain each patch …

embedding embeddings exploration ideas key machinelearning multimodal neighbors project research results text text embedding vit