llava

Definition

A widely copied recipe for giving an LLM sight: take a pretrained vision encoder, add a projection layer mapping its features into the language model's token space, and fine-tune on instruction-following data about images. That last step, visual instruction tuning, is the contribution most often credited to it, and it became the default template for open multimodal models. Because it reuses pretrained parts, it is cheap relative to training multimodally from scratch.