#architecture
15 articles
01
·Large Language Models·FREE·PAPER·12 min read
Mamba and State Space Models — Handling Sequences Without Attention
Attention keeps everything and re-reads it on every step, and the price is quadratic cost in sequence length. State space models take the opposite bet: keep one fixed-size note and update it. Starting from a continuous-time linear system, we trace why S4 can secretly become a convolution, what exactly Mamba made 'selective' in order to give that convolution up, and where the whole approach loses to attention.
02
·Large Language Models·FREE·PAPER·12 min read
Mamba and State Space Models — Handling Sequences Without Attention
Attention keeps everything and re-reads it on every step, and the price is quadratic cost in sequence length. State space models take the opposite bet: keep one fixed-size note and update it. Starting from a continuous-time linear system, we trace why S4 can secretly become a convolution, what exactly Mamba made 'selective' in order to give that convolution up, and where the whole approach loses to attention.
03
·How Transformers Work·★ MEMBER·PAPER·10 min read
Paper Explained: Why Gated DeltaNet Survives 4-Bit Quantization — NVFP4 W4A4 in a Hybrid 27B
A walkthrough of the paper that tested — and overturned — the belief that the recurrent half of a hybrid LLM is too fragile for 4-bit quantization, and explained mechanistically why log-space gates and the delta rule erase quantization noise.
04
·How Transformers Work·★ MEMBER·PAPER·10 min read
Paper Walkthrough: Stop Anchoring to Frame One — Scal3R's Multi-Reference Relative Pose Query
Long videos make online 3D reconstruction collapse. This paper traces the cause to extrapolation against a fixed first-frame anchor and fixes it by adding roughly 1% of trainable tokens to a frozen backbone. Explained from zero.
05
·Paper Deep-Dives·★ MEMBER·PAPER·11 min read
Paper Walkthrough — RoboTok: Mining the Web for Demonstrations That Move Like Yours
A ground-up walkthrough of RoboTok (arXiv:2609.03199), a data engine that pulls manipulation-relevant human demonstrations out of web video. Covers how DTW defines 'similar motion', how that ranking is distilled into an embedding for inner-product search, and what the results do and don't show.
06
·How Transformers Work·★ MEMBER·PAPER·16 min read
Paper Walkthrough: Designing Qwen3.8-Next — Accuracy, Efficiency and Stability as One Problem
A ground-up read of the Qwen3.8-Flash-Next design report: the GDN hybrid, Qwen Sparse Attention, the Gated Residual and n-gram embeddings, judged the way the paper judges them — on loss, on cost, and on stability.
07
·Paper Deep-Dives·★ MEMBER·PAPER·12 min read
Paper Walkthrough — J-Zero: Growing the Challenger, the Solver, and the Judge Together from Zero Data
A model that writes its own problems, solves them, and grades them — with no external data and no human labels. J-Zero's twist is that the grader learns too, which is what lets it keep improving past the two-iteration wall where prior methods stall.
08
·Engineering Process·★ MEMBER·9 min read
Architecture Decisions — Telling Apart What You Can Undo From What You Cannot
How to separate the design decisions you can cheaply reverse from the ones you can never take back. Measuring reversibility, writing ADRs that record what you gave up, and a template for putting trade-offs into words — from first principles.
09
·Paper Deep-Dives·★ MEMBER·PAPER·13 min read
Paper Walkthrough: Turning Game Development into a Verifiable Trajectory Data Engine — RLHEV and AWoMo
A paper arguing that what world models lack is not data or compute but cheap grading. We work through RLHEV — post-training on game-engine checks fused with developer accept/reject decisions — from first principles to equations, interactive figures, and the reported numbers.
10
·Paper Deep-Dives·★ MEMBER·PAPER·11 min read
Paper walkthrough: Apodex 1.1 — scaling agents around completed work
Not a bigger model and not more thinking time — Apodex 1.1 scales two other surfaces: the environments an agent learns in, and the way work is organised across agents. A walkthrough from the task contract to the AgentOS delivery gate, the numbers, and the limits.
11
·How Transformers Work·FREE·PAPER·10 min read
The Transformer, End to End — One Token's Journey from Embedding to Output
Between typing a prompt and getting a word back, a single token gets handed a vector, rewritten in room after room, and finally turned back into language. Tokenizing, embedding, position, attention, feed-forward, residuals, and the output head — walked as one continuous trip rather than a pile of parts.
12
·Paper Deep-Dives·★ MEMBER·PAPER·13 min read
Paper Deep Dive — Large Discovery Models: giving an LLM a value signal for what to try next
An LLM can propose candidates but cannot reliably grade its own proposals. Large Discovery Models tilts the LLM's proposal distribution by an acquisition value drawn from a Gaussian process surrogate, so search keeps working when every evaluation is expensive. From the equations to the reported numbers.
13
·Paper Deep-Dives·★ MEMBER·PAPER·11 min read
Paper Walkthrough: WorldClaw — Agents That Build Walkable, Editable 3D Open Worlds from a Single Sentence
Type 'a tropical pirate stronghold' and get a large-scale 3D world you can walk through — with every building still an editable, standalone asset. A walkthrough of Tencent Hunyuan's WorldClaw, from its plan → terrain → regional-objects pipeline to the render-and-inspect loops where agents check their own work.
14
·Paper Deep-Dives·★ MEMBER·PAPER·9 min read
Mixture of Experts (MoE) from Scratch — Routing and Load Balancing in the Switch Transformer
A ground-up explanation of Mixture of Experts, the sparse architecture behind today's largest LLMs, built strictly from the Switch Transformer paper: the router math, expert capacity, the load-balancing loss, and the three tricks that make sparse training stable.
15
·Paper Deep-Dives·★ MEMBER·PAPER·8 min read
Paper Deep Dive: AskChem — Changing the Unit of Search from Papers to Provenance-Carrying Claims
An NYU-led systems paper that swaps the retrieval unit of chemistry literature search from papers to claims grounded by a DOI and a verbatim quote. We walk through the 2.4M-claim index, evidence graph, and hybrid search design, the evaluation that took DOI resolution from 88.3% to 100%, and the paper's honest limitations.