Do Transformers Actually Work on Time Series? — The Argument and the Practical Answer

In 2022 a single linear layer beat the whole crop of time-series Transformers on the standard benchmarks. The culprit turned out to be tokenization, not attention — a diagnosis PatchTST fixed by patching and iTransformer by transposing the axes. And yet gradient boosting keeps winning the practitioner competitions, for reasons that come down to the shape of real data.

Are Transformers Effective for Time Series Forecasting?

The day a single linear layer won

Transformers conquered language, moved into vision, then into audio. Time series looked like the obvious next stop. Between 2021 and 2022 a wave of time-series-specific variants — Informer, Autoformer, FEDformer — pushed the standard benchmark records forward year after year.

Then, in 2022, came a paper with a title that pulled no punches: "Are Transformers Effective for Time Series Forecasting?" The challenger the authors put up had no attention and no stacked blocks — just a single linear map from input to output. It beat the Transformer models of the day on most of the standard benchmarks.

What makes this interesting is that it is not a story about deep learning losing. The thing at fault was not attention but the way time series were being chopped into tokens before being handed to a Transformer. And even after the right way was found, gradient boosting keeps winning the competitions that look like real work. Those are the three acts of this article.

Why it looked promising in the first place

The motivation was sound. The classic weaknesses covered in time-series forecasting from scratch read like a list of self-attention's strengths, inverted. The distant past is out of reach — a recurrent network passes memory forward one step at a time, so information 365 steps back has thinned out, while attention connects any two positions in a single dot product regardless of distance. Training cannot be parallelized — a recurrent model cannot start step $t+1$ until step $t$ finishes, whereas attention processes every position at once. Cross-variable interactions are hard to write by hand — when temperature, price, inventory and day-of-week all interact, attention looks like it could learn which drives which.

The rebuttal: one linear layer is enough

DLinear is almost anticlimactically simple. Smooth the input with a moving average to pull out a trend component, subtract it to get the seasonal remainder, then apply one matrix to each — mapping an input of length $L$ to an output of length $T$ — and add them.

$x_{\text{trend}}$ is the smooth component from the moving average, $x_{\text{seasonal}}$ is what is left after subtracting it, and each $W$ is a $T \times L$ weight matrix. All the formula says is: give each of the past $L$ points a fixed weight and add them up to produce each future point. No nonlinearity, no gates, no attention. The learned parameters number $2 \times T \times L$ — two or three orders of magnitude below the Transformers it was compared against.

So why was that enough? Three explanations.

1. Self-attention does not see order to begin with. It takes every pairwise dot product and forms a weighted average; before positional encodings are added, permuting the input just permutes the output. In language that is fine, because words carry meaning on their own and order is a refinement. In a time series, order is nearly all of the information: the set {12, 15, 19} means nothing, and "it climbed 12 → 15 → 19" is the whole content. Adding order as an afterthought is a poor fit for this kind of data.

2. A single timestep is far too thin to be a token. The essence of attention is measuring semantic closeness through the dot product of a Query and a Key. But a single observation is one scalar. Project it into a few dozen dimensions and the only thing that grew is the apparent dimensionality, so dot products between such vectors measure noise more than similarity.

And when the scores carry no real differences, look at what softmax does. It turns differences into proportions, so flat scores produce something close to a uniform distribution. Mixing every timestep with equal weight is a moving average. Stack as many layers as you like: if each attention layer is effectively recomputing a moving average, losing to a linear model that contains one explicitly is not surprising.

3. The data is three orders of magnitude short. Even the longer benchmark series run to tens of thousands of points, against the trillions of tokens a language model reads. To win on expressive capacity you need enough data to fill it. In the small-data regime a simple model with a strong inductive bias — a built-in structural assumption — has the advantage, and "split into trend and seasonality, then add linearly" is a rather good assumption about time series.

The real problem was the unit of tokenization

It is worth remembering how vision handled this. Nobody built a Vision Transformer by treating each pixel as a token: the token count would explode, and a single pixel value carries no information about "cat-ness." Only once pixels were grouped into 16×16 patches did tokens become units of meaning and dot products start meaning something. The same thing had been happening with time series — treating one timestep as one token was the same mistake as treating one pixel as one token.

PatchTST: cut into patches, keep the variables apart

PatchTST, from 2023, is the direct prescription for that diagnosis. It has exactly two components.

Patching. Take an input of length $L$, slide a window of length $P$ across it with stride $S$, and make each group of $P$ values a single token. The token count is

where $L$ is the input length, $P$ the patch length, $S$ the stride and $N$ the resulting token count. It is nothing more than counting how many pieces you get when you slide a width-$P$ window across a length-$L$ sequence in steps of $S$.

Two effects arrive together. First, tokens become meaningful: a patch contains a local shape — rising, cresting, a step change — so a dot product between patches actually measures whether two shapes are alike. Second, cost: self-attention scales with the square of the token count, so cutting the token count by a factor of $S$ cuts compute and memory by $S^2$.

There is a second reason this mattered. One of the DLinear paper's observations was that the Transformers of the day did not improve when given a longer look-back window — sometimes they got worse. The history was being shown to them and not used. Patching keeps the token count in check while the look-back grows, which lifts that constraint. The advantage Transformers were supposed to have on long-horizon forecasting only became real at this point.

Channel independence. The second component runs against intuition. Given seven sensors, the natural move is to bundle all seven into one token and let attention learn the cross-variable interactions. PatchTST does the opposite: each variable goes through the same shared-weight model on its own, with no attention across variables. It looks like discarding information, but there are two gains. Seven series become seven training examples for one shared set of weights, so the effective amount of training data grows. And the model stops overfitting to spurious cross-variable correlations — among seven sensors there will always be a pair that happens to move together, with no guarantee the relationship survives into the test period.

One implementation detail is load-bearing: per-window normalization, subtracting each input window's mean and dividing by its standard deviation, then undoing that on the output. Levels drift, so you strip the level off each window and learn only the shape. Crucially the statistics come from inside that window alone, not from the training period — which doubles as leakage prevention.

iTransformer: transpose the axes

iTransformer, from 2024, swaps rows and columns. Until then the token was a timestep (or a patch) and attention ran along the time axis. iTransformer makes the entire series of a single variable into one token: seven variables, seven tokens. Attention now runs between variables, while the time axis is handled by the feed-forward layer inside each token.

This looks like the exact opposite of PatchTST's "drop cross-variable attention," but at root the two agree. Both abandoned "one timestep = one token." Re-cutting tokens into meaningful chunks is the substance; whether those chunks run along the time axis or the variable axis depends on the data.

And yet the practitioner competitions go to gradient boosting

Everything above is a contest fought on academic benchmarks. Move to competitions that resemble real work and the picture changes. In the 2020 M5 competition (daily demand forecasting for Walmart store × item), most of the top solutions were gradient boosting built around LightGBM. In M4 in 2018 the winner was a hybrid of exponential smoothing and a recurrent network, and second place learned weights over multiple methods using gradient boosting — no pure deep model won on its own.

The reason is that real time series have a different shape from benchmark ones. Series are short and numerous — store × SKU runs to tens of thousands or millions of series with a few hundred points each, whereas benchmarks are a handful of very long series, so they measure a different ability. Exogenous variables dominate the outcome — price, markdowns, promotions, holidays, weather. Most of the variation comes from there, while most benchmarks predict the future from the series' own past alone. Zeros are everywhere, and for intermittent demand a squared-error design aiming at the mean does not fit. And there is a hierarchy (SKU → category → store → company), so totals across levels have to be reconciled.

Gradient boosting meets that shape head-on. Put lags, rolling statistics, price, calendar fields and category IDs into columns and it becomes tabular regression; missing values and categoricals are handled directly; a tree naturally expresses a nonlinear step like "demand jumps once price drops below a threshold." Training is fast, contributions are inspectable, and adding an exogenous variable costs one more column.

How to read the benchmark tables

Long-horizon forecasting has effectively standardized on ETT (transformer oil temperature), Electricity, Traffic and Weather crossed with horizons of 96/192/336/720. Look-back windows are not always matched — the optimal input length differs by model, so a comparison that gave one side a short window is meaningless. Implementation details move the numbers — several of the standard codebases drop the final partial batch in the test dataloader, which makes the evaluated sample count depend on batch size, and the choice of which period the normalization statistics come from, or whether there is only one split, moves them too. Recent benchmark-reconstruction papers keep pointing this out, so the safe reading is: believe differences of an order of magnitude, distrust ranking fights in the third decimal. Be suspicious of weak baselines — a table without seasonal naive (emit the value one period ago) and a plain linear model earns skepticism on that basis alone.

How this plays out on the job

The people who need this judgment call work on retail and manufacturing demand forecasting, power and traffic load forecasting, cloud capacity planning, and financial risk estimation. When someone says "a new Transformer came out, let's try it," being able to explain what you try and in what order is the job.

The order is largely fixed. Start with seasonal naive and a moving average as a baseline; if you cannot beat that, nothing downstream matters. Then: few series, long history, few exogenous variables → classical methods (SARIMAX, ETS). Many series where exogenous variables matter → gradient boosting. Many series and a long horizon you want emitted in one shot → deep models. You need a rough read quickly → a time-series foundation model for a zero-shot stalking horse. That ordering maps one-to-one onto the data-shape questions above.

The knobs are worth memorizing. On the PatchTST side: patch_len, stride, context_length or input_size (the look-back), d_model, n_heads, and whether per-window normalization is on. Implementations live in neuralforecast, darts and GluonTS, and in all of them h (forecast horizon) and input_size bite first. On the boosting side: LightGBM's objective (tweedie for intermittent demand, quantile for quantiles), num_leaves, min_data_in_leaf, and declaring categorical features.

The accidents happen around the model rather than inside it.

Comparing without matching look-back windows. When you check "does PatchTST beat DLinear" on your own data, running one side with a short window flips the conclusion. Tune both near their own best before comparing.

Fitting normalization outside the model over the full period. Per-window normalization is closed inside its window and is safe, but running a StandardScaler over the entire dataset beforehand lets test-period statistics flow into training. This is the least visible form of leakage there is.

Reading channel independence as "variables can be thrown away." It is an empirical finding that cross-variable correlations were not dependable on those benchmarks. It is not a reason to discard a relationship with clear causality, like price → demand. Those belong in the model as explicit exogenous variables, not as something attention is expected to rediscover.

Building long horizons by recursing a one-step model. You feed the model its own errors and they accumulate. That is why long-horizon models are direct multi-step (multi-output), and why the training horizon should match the production horizon.

Design reviews and interviews converge on three questions: what the DLinear argument concluded, why channel independence helps, and what you try first in practice. The reasoning for each is the summary below.

Summary

• In 2022 a single linear layer beat the time-series Transformers of the day on the standard benchmarks. The cause was not attention but the choice of one timestep as one token
• A single timestep is one scalar, too thin for dot-product scores to differentiate. Softmax flattens and the attention layer degenerates into a moving average
• PatchTST gave tokens meaning through patching while cutting the token count, which finally made longer look-back windows usable. Channel independence raises effective data volume through weight sharing and suppresses overfitting to spurious correlations
• iTransformer transposed the axes and put attention between variables. The conclusion looks opposite, but both stopped treating one timestep as one token
• In practice you try seasonal naive first, then gradient boosting with exogenous variables; deep models once the series count and horizon line up

Next we look at time-series foundation models — pretrain one model, feed it a series it has never seen — and how much of this work they can actually take over.