2026-08-13·★ MEMBER·PAPER·9 min read

Object Detection from Scratch (from YOLO to DETR)

A from-zero guide to object detection: the classic toolkit of two-stage detectors, anchors, and NMS — and how DETR reframed the whole problem to make all of it unnecessary. Primary source: the DETR paper.

#object-detection #cnn #transformer #detr #computer-vision

ModalityimageTaskdetection

End-to-End Object Detection with Transformers

Primary source — what this article is built on

undefined2020-05-26→undefined2026-08-136y 3mo later

End-to-End Object Detection with TransformersNicolas Carion, Francisco Massa, Gabriel Synnaeve et al. · 2020-05-26 · v3arXiv:2005.12872Paper page·PDF

undefined

We present a new method that views object detection as a direct set prediction problem. Our approach streamlines the detection pipeline, effectively removing the need for many hand-designed components like a non-maximum suppression procedure or anchor generation that explicitly encode our prior knowledge about the task. The main ingredients of the new framework, called DEtection TRansformer or DETR, are a set-based global loss that forces unique predictions via bipartite matching, and a transformer encoder-decoder architecture. Given a fixed small set of learned object queries, DETR reasons about the relations of the objects and the global image context to directly output the final set of predictions in parallel. The new model is conceptually simple and does not require a specialized library, unlike many other modern detectors. DETR demonstrates accuracy and run-time performance on par with the well-established and highly-optimized Faster RCNN baseline on the challenging COCO object detection dataset. Moreover, DETR can be easily generalized to produce panoptic segmentation in a unified manner. We show that it significantly outperforms competitive baselines. Training code and pretrained models are available at https://github.com/facebookresearch/detr.

Not "what is this photo?" but "what is where?"

Image classification returns one label per image: "this is a dog." Object detection goes one step further and answers what is where, and how many. The output is a collection of bounding boxes with labels — "a dog in this box, a person in that box, a car over there" — and the number of boxes changes from image to image.

Think of a group photo. Classification says "this is a class photo." Detection circles every friend in the picture and writes their name next to each circle. You don't know in advance how many people are in the shot, yet you must circle all of them — no more, no less. This "output a set of unknown size" property is what makes detection genuinely hard.

The DETR paper — the primary source for this article — attacks exactly this point. It argues that modern detectors solve the set prediction task indirectly: they define surrogate regression and classification problems on top of proposals, anchors, or grids of object centers, then massage the answers into a set with postprocessing (§1). Let's first look at that indirect approach.

Lineage 1: two-stage detectors — propose, then inspect

Two-stage detectors, with Faster R-CNN as the flagship, first generate a large batch of candidate boxes (proposals) for "places where something might be," then inspect each candidate: is this a dog, and how should the box be nudged? It's a skim-then-study pipeline; in the paper's taxonomy, these methods "predict boxes with respect to proposals" (§2.3).

Lineage 2: one-stage detectors and anchors — tile the image with draft boxes

One-stage detectors such as YOLO and SSD skip the proposal step and predict directly from a grid over the image. Their signature tool is the anchor: a set of draft boxes of various sizes and aspect ratios tiled across the image in advance. Each anchor is asked "is there an object here?" and "if so, how much should the box shift?" Correcting a draft is easier to learn than drawing from a blank page (in the paper's taxonomy: predictions made with respect to anchors or grids of possible object centers, §2.3).

The paper adds a warning, though: the final performance of these systems is heavily influenced by exactly how those initial guesses are set — the design of the anchor set, and the hand-written rules that assign ground-truth boxes to anchors (§1, §2.3).

NMS — the postprocessing step that deletes duplicates

Draft-based prediction has a side effect: several neighboring anchors or grid cells end up predicting nearly identical boxes for the same object. Enter Non-Maximum Suppression (NMS): keep the highest-scoring box, delete every box that overlaps it heavily, repeat. The paper counts NMS as yet another hand-designed component that shapes final performance (§1).

To sum up, the classical detection pipeline drafts with anchors (prior knowledge) and cleans up with NMS (postprocessing). DETR's question is simple: do we actually need the draft and the cleanup?

The dot product — the one tool DETR relies on

To understand DETR's answer you need exactly one tool: the dot product, which measures how closely two vectors point in the same direction. Transformer attention uses this score to decide how much to listen to each piece of information. For the full mechanics see Attention from Scratch; here, just get a feel for it.

FIG 1Rotate the two vectors and watch the dot product change — large when they align, zero when perpendicular. Inside DETR, this similarity score becomes the attention weight between features

DETR's reframing — solve set prediction directly

DETR (DEtection TRansformer) can be stated in one line: treat detection as a direct set prediction problem, and throw away both anchors and NMS. The paper says two ingredients make this possible (§3):

A bipartite-matching loss that forces a strict one-to-one pairing between predictions and ground truth
An architecture that reasons about all objects and the global image context together, emitting every prediction in parallel, in a single pass