What Does AI Actually Learn? Machine Learning with One Straight Line (Tech Explained from Zero #1)

We work out what 'learning' means using nothing but five points and one straight line: a loss that turns misses into a single number, gradient descent down the loss valley, what happens when the learning rate is too big, and a curve that scores perfectly on the data yet fails everywhere else. Every number can be checked by hand. With a companion video.

About this series — Tech Explained from Zero is built so that each part stacks on the previous one: read (or watch) from #1 in order. Every part has a companion video with the same content. Part 1 assumes no background.

From writing rules to showing examples

Say you want software that spots spam. In ordinary programming, a person writes rules: "if the subject contains 'You won', it's spam." But spammers change their wording — "You've won", "Y0u won" — and the rules never stop growing.

Machine learning flips this around. Instead of writing the rule, you show the machine many examples together with the answers, and let it build the rule itself.

So what does "the machine builds the rule" mean concretely? In this article we compute it all the way through, using just five points and one straight line.

The example: study time and quiz scores

Here are five students' study hours and scores on a quiz out of 20 (this is made-up data for the explanation).

We want a rule that predicts the score from the study time. The simplest rule to try is a straight line:

$$\hat{y} = w\,x$$

In words, this is a guess that the score is proportional to the study time.

One symbol at a time: $x$ is the study time (the input), and $\hat{y}$ ("y-hat") is the predicted score — the hat tells it apart from the true score $y$. $w$ is the slope of the line: how many points one extra hour is worth.

This $w$ is the knob we want the machine to set. The technical name is a parameter. With $w=1$, an hour is worth 1 point; with $w=3$, 3 points. Turning the knob changes the slope of the line.

(To keep things simple, we only consider lines through the origin — "0 hours means 0 points". We come back to adding an intercept at the end.)

"Learning" means finding the value of $w$ that fits the data best. But how do we measure "fits best"?

One number for how far off we are: mean squared error

Try $w=1$ first. The predictions are $\hat{y} = 1, 2, 3, 4, 5$ points. The differences (true − predicted) are

$$2-1=1,\quad 5-2=3,\quad 5-3=2,\quad 7-4=3,\quad 11-5=6$$

We want to boil these five differences down to one number, because with a single number we can say "better" or "worse".

What if we just add them up? That is a trap. Try $w=2$: the predictions are $2, 4, 6, 8, 10$, and the differences are

$$0,\quad 1,\quad -1,\quad -1,\quad 1$$

which add up to 0. The line misses, yet we would call the error zero — the positive and negative misses cancel out.

So we square each difference first, add them, and divide by the count to get an average. This is the mean squared error (MSE):

$$L = \frac{1}{N}\sum_{i=1}^{N}\left(y_i - \hat{y}_i\right)^2$$

In words: square each student's miss, then take the average over everyone.

One symbol at a time: $L$ is the loss — the number for how far off we are; $N$ is the number of data points (5 here); $y_i$ is student $i$'s true score and $\hat{y}_i$ the prediction; $\sum$ means "add up over all $i$ from 1 to $N$".

Plugging in numbers:

• $w=1$: squared differences $1, 9, 4, 9, 36$. Total 59, divided by 5: $L = 11.8$
• $w=2$: squared differences $0, 1, 1, 1, 1$. Total 4, divided by 5: $L = 0.8$

$w=2$ is far smaller, so the $w=2$ line fits the data much better. Squaring also counts big misses more heavily: a miss twice as large costs four times as much.

The loss landscape: how L changes as you turn w

Computing $L$ for several values of $w$ gives:

A symmetric valley with its bottom at $w=2$. In fact, in this example, expanding the formula shows that $L$ is exactly a parabola:

$$L(w) = 11\,(w-2)^2 + 0.8$$

In words: the further $w$ moves from 2, the more the loss grows, in proportion to the square of that distance.

(The mean of $x^2$ is 11, the mean of $x\cdot y$ is 22, and the mean of $y^2$ is 44.8, so $L = 11w^2 - 44w + 44.8 = 11(w-2)^2 + 0.8$.)

Two things can be read off this formula. At $w=2$, $(w-2)^2 = 0$, so the loss reaches its minimum of 0.8. And it can never go below 0.8: no straight line passes exactly through all five points.

So now learning can be restated as finding the lowest point of the loss valley.

Walking downhill: gradient descent

Because we have the formula, we could see at a glance that the bottom is at $w=2$. Real AI models, though, have millions or billions of knobs, and nobody can see the whole valley. All you can see is the slope where you are standing.

So you repeat two steps:

1. Measure the slope where you are
2. Move $w$ a little in the opposite direction of the slope (downhill)

This is gradient descent. As a formula, it is one line:

$$w_{\text{new}} = w_{\text{old}} - \eta \cdot \frac{dL}{dw}$$

In words: from where $w$ is now, move against the slope, by an amount proportional to how steep the slope is.

One symbol at a time: $\frac{dL}{dw}$ is the rate at which $L$ grows when you increase $w$ a little — the slope (the derivative). $\eta$ ("eta") is the learning rate, which sets the size of each step. The minus sign makes you go downhill: if the slope is positive (rising to the right) you decrease $w$; if it is negative you increase $w$.

Differentiating the parabola gives the slope for this example:

$$\frac{dL}{dw} = 22\,(w-2)$$

At $w=0$ it is $22 \times (0-2) = -44$: a steep slope falling to the right.

Let's walk with learning rate $\eta = 0.02$, starting from $w=0$:

Step 1 is $0 - 0.02 \times (-44) = 0.88$, and the loss drops from 44.8 to 14.60 in one go. The slope gets gentler near the bottom, so the steps shrink on their own.

Look one level deeper and you can see exactly what each step does. Putting the slope into the update gives

$$w_{\text{new}} - 2 = (1 - 22\eta)\,(w_{\text{old}} - 2)$$

so the gap to the bottom is multiplied by $(1-22\eta)$ every step. With $\eta = 0.02$ that is $1 - 0.44 = 0.56$. The gap goes $-2 \to -1.12 \to -0.627 \to \cdots$, and after 50 steps $w$ is essentially 2.0.

The learning rate: what happens if it's too big

Do bigger steps get you there faster? Check with "the gap is multiplied by $(1-22\eta)$":

• $\eta = 0.05$: the factor is $1 - 1.1 = -0.1$. $w$ goes $0 \to 2.2 \to 1.98 \to 2.002$. The minus sign means it jumps over the bottom to the other side, but the gap shrinks tenfold each time and settles quickly.
• $\eta = 0.1$: the factor is $1 - 2.2 = -1.2$. $w$ goes $0 \to 4.4 \to -0.88 \to 5.456 \to -2.147$. Each jump widens the gap by 1.2×, and $w$ is thrown out of the valley (divergence).

The boundary is where the factor hits $-1$: $\eta = 2/22 \approx 0.091$. With any larger learning rate, this example never reaches the bottom. With a tiny one, it takes thousands of steps. The learning rate is one of the first numbers people tune when training real AI models.

Even with millions of knobs, the procedure is the same: measure the slope for each knob (all of them lined up together are called the gradient) and nudge every knob a little downhill. The one-line formula above sits at the heart of large-scale training.

In code

Everything so far takes a few lines of Python:

import numpy as np

x = np.array([1, 2, 3, 4, 5.0])   # study hours
y = np.array([2, 5, 5, 7, 11.0])  # scores

w = 0.0                            # starting point
for step in range(50):
    grad = 2 * np.mean((w * x - y) * x)   # slope dL/dw
    w -= 0.02 * grad                      # step downhill, learning rate 0.02
print(round(w, 4))                 # → 2.0

The grad line is the derivative of the mean squared error with respect to $w$, $\frac{2}{N}\sum(\hat{y}_i - y_i)\,x_i$, written out directly. At $w=0$ it evaluates to $-44$, matching step 1 in the table.

Memorizing is not understanding

Finally, let's build a rule that fits the data "even better": not a line, but a curve that passes exactly through all five points (a quartic):

$$\hat{y} = -\tfrac{5}{24}x^4 + \tfrac{35}{12}x^3 - \tfrac{331}{24}x^2 + \tfrac{325}{12}x - 14$$

In words, this is a curve with five knobs instead of one, tuned so that it passes through every one of the five students' points.

Since it passes through every point, its loss on the five students is 0 — supposedly "better" than the line's 0.8.

But compute what it predicts where we have no data:

• Someone who studied 0 hours: the line says 0 points, the curve says −14 points
• Someone who studied 7 hours: the line says 14 points, the curve says 0 points

The curve has simply memorized the five students; it has not captured how study time relates to scores. Shrinking the loss on the data you have is a means, not the goal — the goal is to be right on data you haven't seen yet. This memorizing behavior is called overfitting. How to detect and prevent it is the topic of the next part.

What about an intercept?

We only used lines through the origin. If you add an intercept $b$, making $\hat{y} = wx + b$, there are two knobs and the loss valley becomes a bowl (a two-dimensional valley) instead of a parabola. The procedure doesn't change: measure the slope for $w$ and for $b$, and step downhill. For this data, the bottom of the bowl is also at $w=2,\ b=0$ (it happens to be data whose best line goes through the origin).

Using it in practice

The procedure in this article (pick a loss → measure the slope → step downhill by the learning rate → check on data you didn't train on) is exactly what data scientists and machine learning engineers do when they build the first baseline model for a new prediction problem. Before moving to a more complex model, measuring how far a straight line (linear regression) can push the loss down gives you something to compare against, so you can tell whether the later model is actually better.

Names you will actually touch

• scikit-learn's LinearRegression: the fit_intercept argument defaults to True (with an intercept). Set it to False only when you want a line through the origin, as in this article
• scikit-learn's SGDRegressor: fits a linear model by gradient descent. The step size is set by eta0, and how it shrinks over time by learning_rate
• PyTorch: the loss is torch.nn.MSELoss() (it averages by default — the same $L$ as here), and the update is torch.optim.SGD(params, lr=...). lr is this article's $\eta$
• NumPy's numpy.polyfit(x, y, deg): raise deg and you can build a "passes through every point" curve like this article's quartic in one line

Pitfalls that cause real accidents

• Changing the input's units alone can make training diverge. The breaking point $\eta \approx 0.091$ in this example was set by the mean of $x^2$ (11): the boundary is $1/11$. Enter study time in minutes instead of hours and $x$ becomes 60 times larger, the mean of $x^2$ becomes 3600 times larger (39600), and with the same $\eta = 0.02$ the factor is $1 - 2 \times 0.02 \times 39600 = -1583$. It diverges from the very first step. This is why inputs are brought to a common scale (standardized) before training (Efficient BackProp also recommends normalizing the inputs)
• If the loss turns into nan or inf, suspect the learning rate first. When $w$ keeps bouncing and growing, as in the $\eta = 0.1$ example, it eventually exceeds the range numbers can represent
• Don't judge a model by its loss on the data you trained on. With five points, raising the polyfit degree to 4 is enough to drive the training loss to zero. Evaluate on data that was not used for training (how to split it is part #2)

Questions you'll be asked in interviews and on the job

• "The loss isn't going down during training, and partway through it starts rising. What do you check?" — Line of reasoning: first rerun with a smaller learning rate and see whether the loss now falls. If it does, the step was too large for how sharply the valley curves (in this article's terms, the factor $|1 - 22\eta| > 1$). Next, check whether the inputs are on a common scale
• "You built a model with zero error on the training data. Is it a good model?" — Line of reasoning: that alone tells you nothing. Measure it on data not used for training; a large gap from the training error points to overfitting
• "Why square the errors? Why not just add them?" — Line of reasoning: use this article's $w=2$ example to show that plain addition gives 0. Squaring stops misses from canceling out, and counts big misses more heavily

Summary

• Learning means setting the parameters (knobs) so the model fits the example data
• Fit is measured by a loss; for predicting numbers, the basic one is the mean squared error (square the differences, then average)
• The bottom of the valley is found by gradient descent: move a little against the slope, scaled by the learning rate. Too large a learning rate diverges
• Zero loss on your data doesn't mean you'll be right elsewhere (overfitting)

Next (#2): Memorizing vs. understanding — how to split data into training, validation and test sets, and how to catch overfitting.

Go deeper

• What is machine learning? — a map of supervised, unsupervised and reinforcement learning
• Loss functions and optimization — cross-entropy for classification, SGD
• Gradient descent — ravines, saddle points and the ceiling on the learning rate