Time-Series Forecasting from Scratch — From Classical Methods to Foundation Models

What makes forecasting different from ordinary regression is that the rows are not independent and the test set is always in the future. Autocorrelation and stationarity first, then the intuition behind ARIMA, seasonal and trend decomposition, and where deep and foundation models actually sit. The last third is evaluation — no random splits, how to backtest, and why MAPE lies.

What makes this different from ordinary regression

"Predict tomorrow's sales from sales up to today." It looks like plain regression, and any learning library will happily print a number. That validation score will almost certainly be better than production. Time series lack all four properties that ordinary supervised learning quietly assumes.

Rows are not independent. Standard learning assumes samples are drawn independently from the same distribution. In a time series, two adjacent points carry almost the same information. Know today's temperature and you already half-know tomorrow's.

Order carries meaning. Shuffle the rows of an image dataset and nothing is lost. Shuffle a time series and the information itself disappears.

The distribution moves. Mean and variance both drift. Last year's demand level is not this year's demand level.

The future leaks in easily. The moment you engineer a feature like "7-day moving average", you create room for values the model could not have known at prediction time. That accident gets a section of its own at the end.

Autocorrelation: your own past is the strongest feature

Shift a series by $k$ steps, compare it against itself, and the correlation you get is the autocorrelation at lag $k$.

$y_t$ is the value at time $t$, $\mathrm{Cov}$ is covariance (how much two things move together) and $\mathrm{Var}$ is variance. When $\rho_k$ is near 1, looking $k$ steps back tells you most of what today looks like. It is a ratio, which says: how much today and the point $k$ steps back rise and fall together, divided by the series' own spread so the number means the same thing on any scale. Near 0, the point $k$ steps back tells you nothing about now.

Plot this for $k=1,2,3,\dots$ and you have the ACF, where nearly every time-series job begins. Daily retail data spikes at $k=7$ (day of week); monthly data spikes at $k=12$. The periodicity is visible at a glance.

Stationarity: betting that the rules stay the same

Every classical method assumes stationarity — that the mean, variance and autocorrelation structure do not depend on time. The reason is simple: a model estimates a rule from the past and applies it to the future, so if the rule changes midway, the estimate was meaningless.

Real series are rarely stationary. A trend moves the mean, so we take a difference.

That is all it does: replace "the value" with "the change since yesterday". Put in words, $\Delta y_t$ is a new series answering "how much higher is this step than the one before it?" — a question about level becomes a question about movement. A straight-line trend disappears after one pass, a curved one after two. For an annual cycle you use the seasonal difference $y_t - y_{t-12}$.

Do not overdo it. Differencing too often shaves off real signal and amplifies what noise remains. The rule is "only as many times as it takes to become stationary".

ARIMA: three parts, and the name spells them out

ARIMA is a plain model whose name is simply the initials of its three parts.

AR (autoregression) explains today as a weighted sum of your own past.

$\phi_i$ is the weight on the value $i$ steps back, $p$ is how far back you look, $\varepsilon_t$ is the error and $c$ a constant. As a sentence it is a recipe which says: today = a baseline, plus some fraction of yesterday, plus some fraction of the day before, and so on, plus whatever luck today brought. The intuition is inertia — "it was high yesterday, so it is high today".

MA (moving average) is a weighted sum of past errors.

$\mu$ is the mean and $\theta_j$ the weight on the error $j$ steps back. The same thing in words: today = the usual level + today's luck + part of yesterday's luck still echoing + ..., where "luck" is whatever the model could not account for. The intuition is the tail of a shock — "last week's unexpected dip is still working its way through". Confusingly, this is not the smoothing moving average.

I (integration) is the differencing above: difference $d$ times, then fit AR and MA. That completes $\mathrm{ARIMA}(p,d,q)$. Give the seasonal component its own set of terms and you get $\mathrm{SARIMA}(p,d,q)(P,D,Q)_s$, where $s$ is the period.