[P] seemore: Implement a Vision Language Model from Scratch

Hi all, I implemented a vision language model consisting of an image encoder, a multimodal projection module and a decoder language model in pure pytorch. Think of this as a simplified version of what you see in GPT-4 or Claude 3 in terms of vision capabilities demonstrated by a language model (think moondream 2 or LLaVA when it comes to Open Source Models). The name ‘seemore’ is my way of paying homage to Andrej Karpathy’s project ‘makemore’ because here I …

capabilities claude claude 3 decoder encoder gpt gpt-4 image language language model machinelearning multimodal projection pytorch scratch simplified terms think vision vision language model