Do Transformers Actually Work on Time Series? — The Argument and the Practical Answer

In 2022 a single linear layer beat the whole crop of time-series Transformers on the standard benchmarks. The culprit turned out to be tokenization, not attention — a diagnosis PatchTST fixed by patching and iTransformer by transposing the axes. And yet gradient boosting keeps winning the practitioner competitions, for reasons that come down to the shape of real data.

Are Transformers Effective for Time Series Forecasting?

The day a single linear layer won

Transformers conquered language, moved into vision, then into audio. Time series looked like the obvious next stop. Between 2021 and 2022 a wave of time-series-specific variants — Informer, Autoformer, FEDformer — pushed the standard benchmark records forward year after year.

Then, in 2022, came a paper with a title that pulled no punches: "Are Transformers Effective for Time Series Forecasting?" The challenger the authors put up had no attention and no stacked blocks — just a single linear map from input to output. It beat the Transformer models of the day on most of the standard benchmarks.

What makes this interesting is that it is not a story about deep learning losing. The thing at fault was not attention but the way time series were being chopped into tokens before being handed to a Transformer. And even after the right way was found, gradient boosting keeps winning the competitions that look like real work. Those are the three acts of this article.

Why it looked promising in the first place

The motivation was sound. The classic weaknesses covered in time-series forecasting from scratch read like a list of self-attention's strengths, inverted. The distant past is out of reach — a recurrent network passes memory forward one step at a time, so information 365 steps back has thinned out, while attention connects any two positions in a single dot product regardless of distance. Training cannot be parallelized — a recurrent model cannot start step $t+1$ until step $t$ finishes, whereas attention processes every position at once. Cross-variable interactions are hard to write by hand — when temperature, price, inventory and day-of-week all interact, attention looks like it could learn which drives which.

The rebuttal: one linear layer is enough

DLinear is almost anticlimactically simple. Smooth the input with a moving average to pull out a trend component, subtract it to get the seasonal remainder, then apply one matrix to each — mapping an input of length $L$ to an output of length $T$ — and add them.

$x_{\text{trend}}$ is the smooth component from the moving average, $x_{\text{seasonal}}$ is what is left after subtracting it, and each $W$ is a $T \times L$ weight matrix. All the formula says is: give each of the past $L$ points a fixed weight and add them up to produce each future point. No nonlinearity, no gates, no attention. The learned parameters number $2 \times T \times L$ — two or three orders of magnitude below the Transformers it was compared against.

So why was that enough? Three explanations.

1. Self-attention does not see order to begin with. It takes every pairwise dot product and forms a weighted average; before positional encodings are added, permuting the input just permutes the output. In language that is fine, because words carry meaning on their own and order is a refinement. In a time series, order is nearly all of the information: the set {12, 15, 19} means nothing, and "it climbed 12 → 15 → 19" is the whole content. Adding order as an afterthought is a poor fit for this kind of data.

2. A single timestep is far too thin to be a token. The essence of attention is measuring semantic closeness through the dot product of a Query and a Key. But a single observation is one scalar. Project it into a few dozen dimensions and the only thing that grew is the apparent dimensionality, so dot products between such vectors measure noise more than similarity.

And when the scores carry no real differences, look at what softmax does. It turns differences into proportions, so flat scores produce something close to a uniform distribution. Mixing every timestep with equal weight is a moving average. Stack as many layers as you like: if each attention layer is effectively recomputing a moving average, losing to a linear model that contains one explicitly is not surprising.

3. The data is three orders of magnitude short. Even the longer benchmark series run to tens of thousands of points, against the trillions of tokens a language model reads. To win on expressive capacity you need enough data to fill it. In the small-data regime a simple model with a strong inductive bias — a built-in structural assumption — has the advantage, and "split into trend and seasonality, then add linearly" is a rather good assumption about time series.

The real problem was the unit of tokenization

It is worth remembering how vision handled this. Nobody built a Vision Transformer by treating each pixel as a token: the token count would explode, and a single pixel value carries no information about "cat-ness." Only once pixels were grouped into 16×16 patches did tokens become units of meaning and dot products start meaning something. The same thing had been happening with time series — treating one timestep as one token was the same mistake as treating one pixel as one token.