Video Understanding from Scratch — From a Pile of Frames to a Sense of Time

A video is not just a lot of images. This piece splits video understanding into three questions — which frames to take (sampling), how to mix them (temporal attention), and how to shrink a long video (compression) — and works through each with equations, interactive figures, and code.

Is Space-Time Attention All You Need for Video Understanding? (TimeSformer)

A video is not just a lot of images

We already have models that read images, so surely a video is just those images fed in one second at a time. Try it and you hit two walls.

The first is volume. A 30 fps video is 1,800 frames per minute. Push one frame through an image Transformer and you get hundreds of tokens, so a single minute balloons into hundreds of thousands of tokens. No amount of context-length growth catches up with that multiplication.

The second wall — the deeper one — is order. "Putting a cup down" and "picking a cup up" contain exactly the same frames once you dump them into a bag; only the sequence differs. Any method that averages frames into one vector cannot tell them apart, even in principle. Understanding a video is not about naming what is in it. It is about reading how things changed.

So video understanding splits into three questions, and this article takes them in order:

1. Sampling: how many frames, and from where
2. Temporal attention: how the chosen frames get mixed
3. Compression: how far a long video can be squeezed

How a single image becomes tokens is covered in Paper Deep Dive: ViT, and how those tokens attach to a language model in How VLMs Came Together. Here we pick up where the time axis enters.

The analogy: describing a two-hour film over the phone

Imagine summarising a two-hour film for a friend on the phone. There is no time to narrate every shot, so you make three decisions without noticing. Which scenes to pick (one every ten minutes, or only the moments where the plot turns). How to convey the order (tell them out of sequence and the story collapses). How to shrink long stretches (a thirty-minute meeting scene becomes one clause). Sampling, temporal attention and compression are just machine names for those three decisions.

Sampling: how many frames, and from where

The simplest approach, still the most widely used, is uniform sampling: split the whole video into N equal intervals and take one frame from each.

$T$ is the total number of frames, $N$ is how many you want, $t_i$ is the position of the $i$-th frame you take, and $\lfloor \cdot \rfloor$ means round down. In words: whatever the length of the video, carve it into exactly N evenly spaced picks.

The advantage is that the input size does not depend on the video's length. GPU memory becomes predictable and batching works. The cost is that longer videos get coarser spacing: eight frames from a two-minute clip means one frame every fifteen seconds, and a one-second event is almost certainly missed.

Fixed-fps sampling pins the interval instead — one frame per second, say. Fewer misses, but the count now scales with duration. Ten minutes gives 600 frames, and the budget breaks before the model does.

Training adds one more trick: divide the video into N segments and draw a random frame from inside each segment. The same video yields different frames each epoch, which acts as augmentation. At inference you pin the pick to the segment's centre.

A different family is keyframe extraction: compare neighbouring frames and drop the ones where little changed. It works well on mostly static surveillance footage, but it can no longer answer a question whose answer is "nothing happened for three minutes." What you discarded was not redundancy — it was the information about elapsed time.

Temporal attention: how frames get mixed

Once you have N frames, how do you fuse them into one understanding? Historically there have been three generations.