Students Professional
JA EN

#multimodal

17 articles

01 ·Audio & Speech·★ MEMBER·PAPER·12 min read Paper Walkthrough: Motion-Omni — Speaking and Moving in One Forward Pass A walkthrough of Motion-Omni, a framework where a spoken dialogue model emits its spoken reply and its full-body gestures in the same pass. We follow the design that conditions motion on hidden states instead of waveforms, what it cost to train, and what the paper admits it has not shown. 02 ·Audio & Speech·★ MEMBER·PAPER·9 min read Paper Explained: Last Translation Benchmark — Measuring Translation with Breaking Examples and Verification Rules Machine translation benchmarks are saturating, and neither automatic metrics nor human evaluation can be fully trusted. The response: collect human-written examples that break frontier models, and attach handcrafted verification rules to each one. A ground-up reading of Last Translation Benchmark. 03 ·★ MEMBER·PAPER·9 min read Paper Walkthrough: The Design Fundamentals of Pixel Text Representation Learning An encoder that reads meaning straight off the pixels, never converting glyphs to character codes. This EMNLP 2026 paper argues that what decides its quality is not data volume but four design choices — explained from zero. 04 ·★ MEMBER·PAPER·10 min read Paper Walkthrough: Puffin-World — A World Model That Remembers Which Way Is Up What changes when you give an image generator a sense of gravity? A ground-up walkthrough of Puffin-World's Omni-Camera representation and physics propagation, with equations and interactive figures. 05 ·Agents·★ MEMBER·PAPER·13 min read Paper Walkthrough: UI-Venus-2 — Taking Screen-Operating Agents From Benchmarks to Real Work A ground-up read of the UI-Venus-2 Technical Report: a GUI agent that drives phones, browsers and desktops from pixels alone. What it scales, how it verifies its own training data, and where it still falls short. 06 ·★ MEMBER·PAPER·14 min read Paper Explained: VBVR-Pro — Making "Thinking in Pictures" Trainable, Scorable, and Comparable Native visual reasoning treats image and video generation as the medium of thought. VBVR-Pro supplies 300 procedurally generated tasks and deterministic scorers that replace the VLM judge, then pits image, video, and interleaved generation against each other on identical ground. Written for readers starting from zero. 07 ·RAG & Retrieval·★ MEMBER·PAPER·12 min read Paper Walkthrough: WeMM-Embedding — Putting Text, Images and Video on One Ruler A walkthrough of WeMM-Embedding (2B/4B/9B), Tencent's universal multimodal embedding family. The unified pair format, the <embedding> token, Matryoshka dimensions, two-stage training and distillation — explained from scratch, using only the numbers the paper reports. 08 ·Agents·★ MEMBER·PAPER·11 min read Paper Walkthrough: OmniScientist — An AI Scientist That Actually Looks at the Raw Data Automating every step of the research workflow doesn't help much if the only thing the agent ever sees is somebody else's summary table. OmniScientist puts raw waveforms, images and 3-D point clouds in front of the agent for the whole research lifecycle and enforces the gates in Python — walked through from the paper itself, up to the 21.7% finding buried in a seismic benchmark. 09 ·VLMs & Multimodal·★ MEMBER·PAPER·8 min read How VLMs Came Together — Wiring a Vision Encoder into an LLM A vision-language model is three parts: a vision encoder, a projection layer, and an LLM. From the bridge CLIP built, through the three schools of connector design, to resolution strategies and the implementation constraints that bite in practice. 10 ·VLMs & Multimodal·★ MEMBER·PAPER·8 min read How VLMs Came Together — Wiring a Vision Encoder into an LLM A vision-language model is three parts: a vision encoder, a projection layer, and an LLM. From the bridge CLIP built, through the three schools of connector design, to resolution strategies and the implementation constraints that bite in practice. 11 ·Agents·★ MEMBER·PAPER·9 min read Paper Explained: Video-DeepResearch — Agents That Watch a Video, Then Chase Down Every Lead A walkthrough of Video-DeepResearch, which pulls visual clues out of a video and corroborates them with web search. We follow the primary source to see how two failure modes — a modality bias that makes models dodge visual tools, and knowledge leakage that lets them answer from memory alone — are beaten with stage-wise tool unlocking and a two-phase SFT + GRPO recipe. 12 ·★ MEMBER·PAPER·10 min read Paper Deep-Dive: The 'Physics' of Multimodal Pretraining — Which Way Does Knowledge Actually Flow? What happens when a single model learns text and images together from day one? A large-scale controlled study from Meta FAIR and collaborators maps the asymmetric flow of knowledge between modalities, the conditions for synergy, the 'vision laziness' caused by late integration, and a recipe that gets strong image generation from just 5% of the data budget. 13 ·Agents·★ MEMBER·PAPER·10 min read Paper Walkthrough: Qwen-UI-Agent — How Alibaba Built a GUI Agent That Works on Real Phones and PCs A guided tour of Alibaba MAI-UI team's technical report on Qwen-UI-Agent: a foundation GUI agent trained on 100+ physical phones, a hybrid GUI+CLI action space, and online RL over 100+ turn trajectories, reaching 92.2% on a real-device benchmark. 14 ·★ MEMBER·PAPER·8 min read Paper Walkthrough: Hunyuan3D-Buffalo 1.0 — One Brain for 3D Understanding, Generation, and Editing Describe a 3D shape in words, generate it from text, edit it by instruction, carve out a named part — Tencent Hunyuan's Hunyuan3D-Buffalo 1.0 does all four in a single model. We walk through the 87M-sample data factory behind it, the Nano3D-v2 editing-pair pipeline, and the paper's most practical finding: training generation makes editing better. 15 ·Paper Deep-Dives·★ MEMBER·PAPER·8 min read Paper Walkthrough: Alpamayo — NVIDIA's Reasoning Model for Autonomous Driving A first-principles read of NVIDIA's driving VLA Alpamayo-R1: the Chain of Causation dataset that structures reasoning as cause and effect, flow-matching trajectory decoding that fits in 99 ms, and the three-stage training recipe that uses RL to punish gaps between what the model says and what it drives. 16 ·VLMs & Multimodal·★ MEMBER·PAPER·7 min read Paper Deep Dive — CLIP: Putting Words and Images on One Map A reading of the CLIP paper (Radford et al., 2021) grounded strictly in its own text: how the in-batch positives and negatives work, why zero-shot classification is really a hypernetwork trick, the five points that prompt wording is worth, and the long list of limits the authors wrote themselves. 17 ·VLMs & Multimodal·★ MEMBER·PAPER·7 min read Paper Deep Dive — CLIP: Putting Words and Images on One Map A reading of the CLIP paper (Radford et al., 2021) grounded strictly in its own text: how the in-batch positives and negatives work, why zero-shot classification is really a hypernetwork trick, the five points that prompt wording is worth, and the long list of limits the authors wrote themselves.