VLMs & Multimodal
CLIP, ViT, and putting images and language in one space
01
·VLMs & Multimodal·★ MEMBER·PAPER·7 min read
Paper Deep Dive — ViT: Treating an Image Like a Sentence
A reading of the ViT paper (Dosovitskiy et al., 2020/2021) grounded strictly in its own text: the move of treating 16x16 patches as words, what the position-embedding ablation actually showed, the price of dropping the convolutional inductive bias, and how conditional the claim 'beats CNNs at scale' really is.
02
·VLMs & Multimodal·★ MEMBER·PAPER·7 min read
Paper Deep Dive — CLIP: Putting Words and Images on One Map
A reading of the CLIP paper (Radford et al., 2021) grounded strictly in its own text: how the in-batch positives and negatives work, why zero-shot classification is really a hypernetwork trick, the five points that prompt wording is worth, and the long list of limits the authors wrote themselves.
03
·VLMs & Multimodal·★ MEMBER·PAPER·8 min read
How VLMs Came Together — Wiring a Vision Encoder into an LLM
A vision-language model is three parts: a vision encoder, a projection layer, and an LLM. From the bridge CLIP built, through the three schools of connector design, to resolution strategies and the implementation constraints that bite in practice.
04
·VLMs & Multimodal·★ MEMBER·PAPER·8 min read
BEV Representations From Scratch — Fusing Multiple Cameras Into One Top-Down Map
How a self-driving car turns six camera feeds into a single top-down map. Starting from perspective projection, we build up to the two big design philosophies: LSS, which pushes features into 3D via a predicted depth distribution, and Transformer-style methods like BEVFormer that pull information with BEV queries.
05
·VLMs & Multimodal·★ MEMBER·PAPER·10 min read
Document AI and OCR Today — How an LLM Ends Up Reading Your Invoices
How machines came to read invoices and scanned PDFs, from first principles: text detection and CTC, how errors are measured (CER), LayoutLM's trick of embedding coordinates alongside words, the OCR-free Donut line, and today's habit of handing the page straight to a VLM — plus the walls that matter in production: tables, handwriting, and hallucination.
06
·VLMs & Multimodal·★ MEMBER·PAPER·8 min read
Video Understanding from Scratch — From a Pile of Frames to a Sense of Time
A video is not just a lot of images. This piece splits video understanding into three questions — which frames to take (sampling), how to mix them (temporal attention), and how to shrink a long video (compression) — and works through each with equations, interactive figures, and code.