vlms

Definition

Vision-language models: systems that take images (or video) together with text and produce text. The common recipe attaches a visual encoder to a language model through a projection layer, so the image arrives as tokens the LLM can attend over. They power visual question answering, chart and document reading, and GUI agents. Distinguished from vision-only models, which classify or detect, and from text-only LLMs.

Explainers using this term

• Paper Walkthrough: GST-Bench — Can VLMs Build a Global Map of a Scene from Video?GST-Bench: Can VLMs Develop Global Spatial Awareness from Video?
• Paper explained: DEFT-RLVR — show a driving VLM the future trajectory too early and it fabricates its reasoningDeferred Exposure of Future Trajectories for Verifiable Reasoning in Autonomous Driving VLMs

In the textbook

• How VLMs Came About, and Where They Are