Time-Series Forecasting from Scratch — From Classical Methods to Foundation Models

What makes forecasting different from ordinary regression is that the rows are not independent and the test set is always in the future. Autocorrelation and stationarity first, then the intuition behind ARIMA, seasonal and trend decomposition, and where deep and foundation models actually sit. The last third is evaluation — no random splits, how to backtest, and why MAPE lies.

What makes this different from ordinary regression

"Predict tomorrow's sales from sales up to today." It looks like plain regression, and any learning library will happily print a number. That validation score will almost certainly be better than production. Time series lack all four properties that ordinary supervised learning quietly assumes.

Rows are not independent. Standard learning assumes samples are drawn independently from the same distribution. In a time series, two adjacent points carry almost the same information. Know today's temperature and you already half-know tomorrow's.

Order carries meaning. Shuffle the rows of an image dataset and nothing is lost. Shuffle a time series and the information itself disappears.

The distribution moves. Mean and variance both drift. Last year's demand level is not this year's demand level.

The future leaks in easily. The moment you engineer a feature like "7-day moving average", you create room for values the model could not have known at prediction time. That accident gets a section of its own at the end.

Autocorrelation: your own past is the strongest feature

Shift a series by $k$ steps, compare it against itself, and the correlation you get is the autocorrelation at lag $k$.

$y_t$ is the value at time $t$, $\mathrm{Cov}$ is covariance (how much two things move together) and $\mathrm{Var}$ is variance. When $\rho_k$ is near 1, looking $k$ steps back tells you most of what today looks like. It is a ratio, which says: how much today and the point $k$ steps back rise and fall together, divided by the series' own spread so the number means the same thing on any scale. Near 0, the point $k$ steps back tells you nothing about now.

Plot this for $k=1,2,3,\dots$ and you have the ACF, where nearly every time-series job begins. Daily retail data spikes at $k=7$ (day of week); monthly data spikes at $k=12$. The periodicity is visible at a glance.

Stationarity: betting that the rules stay the same

Every classical method assumes stationarity — that the mean, variance and autocorrelation structure do not depend on time. The reason is simple: a model estimates a rule from the past and applies it to the future, so if the rule changes midway, the estimate was meaningless.

Real series are rarely stationary. A trend moves the mean, so we take a difference.

That is all it does: replace "the value" with "the change since yesterday". Put in words, $\Delta y_t$ is a new series answering "how much higher is this step than the one before it?" — a question about level becomes a question about movement. A straight-line trend disappears after one pass, a curved one after two. For an annual cycle you use the seasonal difference $y_t - y_{t-12}$.

Do not overdo it. Differencing too often shaves off real signal and amplifies what noise remains. The rule is "only as many times as it takes to become stationary".

ARIMA: three parts, and the name spells them out

ARIMA is a plain model whose name is simply the initials of its three parts.

AR (autoregression) explains today as a weighted sum of your own past.

$\phi_i$ is the weight on the value $i$ steps back, $p$ is how far back you look, $\varepsilon_t$ is the error and $c$ a constant. As a sentence it is a recipe which says: today = a baseline, plus some fraction of yesterday, plus some fraction of the day before, and so on, plus whatever luck today brought. The intuition is inertia — "it was high yesterday, so it is high today".

MA (moving average) is a weighted sum of past errors.

$\mu$ is the mean and $\theta_j$ the weight on the error $j$ steps back. The same thing in words: today = the usual level + today's luck + part of yesterday's luck still echoing + ..., where "luck" is whatever the model could not account for. The intuition is the tail of a shock — "last week's unexpected dip is still working its way through". Confusingly, this is not the smoothing moving average.

I (integration) is the differencing above: difference $d$ times, then fit AR and MA. That completes $\mathrm{ARIMA}(p,d,q)$. Give the seasonal component its own set of terms and you get $\mathrm{SARIMA}(p,d,q)(P,D,Q)_s$, where $s$ is the period.

Decomposition: trend, season, remainder

The other basic move is decomposition.

$T_t$ is the trend (the long-run level), $S_t$ the seasonal component and $R_t$ the remainder. It is a bookkeeping identity which says an observation is three things added together: a slowly moving level, a pattern that repeats on schedule, and whatever neither of those explains. There is a multiplicative form $y_t = T_t \times S_t \times R_t$ too, and you choose it when the size of the seasonal swing is proportional to the level. Taking logs turns multiplicative into additive, which is why "log-transform first" so often works in practice.

The main use is diagnosis rather than prediction. If an obvious cycle is still sitting in the remainder, the season has not been captured. If the remainder is large only in one stretch, something happened there — a campaign, an outage, missing data. STL is the tool for doing this robustly.

Where deep and foundation models sit

A classical method fits one model per series: ten thousand stores means ten thousand estimated models. The biggest shift deep learning brought was the global model — train a single model across all series and express the differences between them through embeddings and covariates. A series with a short history can then learn from series that look like it.

DeepAR-style models use a recurrent network to emit the parameters of a distribution for the next point, so repeated sampling gives you prediction intervals directly. N-BEATS-style models drop recurrence entirely, stacking fully connected blocks and passing the residual on to the next one. But note: with few series and short histories, deep models routinely lose to classical methods and to naive baselines. The advantage comes from having many series, so with one series the premise does not hold.

Time-series foundation models extend that line: pre-train one model on a large, heterogeneous pile of series and feed it an unseen series as-is. The value is less accuracy than speed to a first number — hand over data and a zero-shot baseline appears in minutes. Granularity and exogenous variables are still yours to design.

Evaluation: this is where things break

Most time-series failures come from the evaluation design, not the model choice.

Never split randomly

Split randomly for $K$-fold cross-validation and you inevitably create folds that train on future points to predict past ones — a situation that never occurs in production. Worse, autocorrelation means adjacent points carrying nearly identical information end up as twins separated across train and validation. The validation score comes out better than reality, always. It is the time-series-specific worst case of the leakage covered in overfitting and evaluation.

Walk forward in time instead

Use an expanding window (advance the origin and grow the training span) or a rolling window (fixed width, slid forward), always in the direction "training span → the validation span immediately after it". That is backtesting. And never decide from a single split: move the origin repeatedly and look at both the mean and the spread.

Two places leakage gets in

If a 7-day moving average is a feature, the feature on the last training day can reach into the validation span. Leave a gap the size of the window plus the forecast horizon between the end of training and the start of validation. The second place is preprocessing: computing standardisation statistics over the whole period is leakage that nobody notices, so fit the scaler on each split's training span only.

Match the horizon, and always keep a baseline

Roll a one-step-ahead model 24 times to reach 24 hours out and you keep feeding it its own errors, so error accumulates. Either train a separate model per horizon or make the model multi-output — either way, the training horizon must match the production horizon. Then line up the naive forecast (yesterday's value), the seasonal naive (one period back) and a plain mean. Deep models that cannot beat those are not rare.

The MAPE trap

The most used and most misunderstood metric in the field.

$\hat{y}_t$ is the prediction and $n$ the number of evaluated points. Read in words: at each point work out what fraction of the actual you missed by, average that across the points, and multiply by 100 — "off by what percent, typically". Four traps. First, it is undefined when the actual is zero — fatal for intermittent demand, where some days have no sales. Second, the smaller the actual, the more the error is inflated: missing by one unit on a one-unit day is 100%, missing by ten on a thousand-unit day is 1%. Third, it is asymmetric: under-forecasting caps out at 100% while over-forecasting has no ceiling, so optimising MAPE pushes the model low. Fourth, mixing series at different levels breaks the average (more below).

The usual alternative is MASE.

The denominator is the mean error of "just reuse the value $s$ periods ago"; the numerator is how far your own forecast misses on average. So MASE says how many times worse than naive you are: below 1 you win. It is scale-free, so it can be aggregated across series, and it does not break on zeros.

Decide how you aggregate

Average a per-series metric naively and the small series dominate. The ones that matter in revenue terms are the large ones. How you average is a business decision, not a technical one.

How this shows up on the job

The people who build forecasts are doing demand planning in retail and manufacturing, capacity planning for infrastructure, energy and traffic demand, and advertising budget allocation. What they share is that the forecast turns straight into "how many units do we order".

The tooling is fairly settled. On the classical side, statsmodels' SARIMAX(order=(p,d,q), seasonal_order=(P,D,Q,s)) and STL(period=...). For validation, scikit-learn's TimeSeriesSplit(n_splits=..., gap=..., max_train_size=...) — that gap is the buffer described above. Prophet exposes changepoint_prior_scale (how easily the trend bends), seasonality_mode='multiplicative' and holidays. On the deep side, darts and GluonTS give you context_length, prediction_length and freq. Features are usually lags, rolling statistics, Fourier terms and holiday flags.

Filling missing values with zero. "Nothing sold" and "the store was closed" are different events. Fill with zero and the model learns a demand dip that never happened.

Leaving holidays, campaigns and price changes out of the exogenous variables. You will miss the same day in the same direction every year, and the failure gets absorbed into the seasonal component where it is hard to see.

Treating sales as demand. Units sold on a sell-out day measure inventory, not demand. Train on that and you under-estimate demand, order less, and sell out again — the forecast pollutes its own training data.

Shipping point forecasts only. The right inventory level is not average demand; it is a quantile set by the ratio of stockout cost to holding cost.

Three questions come up repeatedly in interviews and reviews. "Why can't you use a random $K$-fold on a time series?" — because folds that predict the past from the future appear, and autocorrelated twins get split across train and validation. "Name the problems with MAPE." — division by zero, inflation at small actuals, asymmetry. "Deep model or ARIMA first?" — seasonal naive first, then classical, then deep if you have many series. Being able to state that order is itself the argument for your design judgement.

Summary

• Time series differ from ordinary regression in two ways: rows are not independent, and the test set is always in the future
• Find the period with autocorrelation, reach stationarity with differencing — and difference the minimum number of times
• ARIMA is AR (inertia) + MA (the tail of a shock) + I (differencing). Deep models are fundamentally global models, and with few series that premise fails
• Evaluate by walking forward, leave a gap, fit preprocessing on the training span only, and always keep a baseline
• MAPE breaks on zeros, asymmetry, and inflation at small actuals

Next we take the same "read a sequence" problem into audio — from waveform to text, inside a speech recogniser.