Decision Trees and Gradient Boosting — Still the Champion on Tabular Data

On data made of rows and columns, ensembles of decision trees are still the first thing to reach for. From the intuition behind a single split to boosting, what XGBoost and LightGBM actually do, and why neural networks have not taken this territory — built up from nothing.

XGBoost: A Scalable Tree Boosting System

Most of the world's data is still a table

The attention goes to models that read images and text, but the data actually moving through companies every day is still mostly rows and columns. One row is one loan application, one transaction, one machine on the floor. The columns are age, amount, hours of operation, error code — different meanings, different units, sitting side by side.

And here is the surprising part: deep learning has not taken this territory. Credit scoring, fraud detection, demand forecasting, click-through prediction — the first thing practitioners reach for is a pile of decision trees. In competitions like Kaggle, GBDT (Gradient Boosted Decision Trees) has stayed a default for years.

Why does a stack of Transformers lose to a technique from the last century? This article starts from the humblest possible mechanism — a single yes/no split — and walks in a straight line to the answer. What is machine learning is enough background; nothing else is assumed.

The metaphor: Twenty Questions

A decision tree is the game of Twenty Questions, verbatim. Someone thinks of a thing, and you narrow it down with yes/no questions alone.

Put that in a loan office. "Is annual income above $50,000?" → yes → "Has the applicant held the job three years or more?" → no → "Are there two or fewer loans elsewhere?" → yes → likely to repay.

That is a decision tree. You walk down from the top through questions (splits) and land in a box (leaf) with the answer written in it. The rules are visible, so you can explain to a person why the decision came out that way. In finance and healthcare, where a reason has to be produced on demand, that property is not a nice-to-have.

Good questions and bad questions exist here too, exactly as in the game. "Is the surname Tanaka?" almost never lands, so it barely narrows anything. A good question is one that leaves the remaining candidates lopsided after you hear the answer. Training a tree is nothing more than finding good questions, automatically, over and over.

The mechanism: putting a number on "good split"

So how do you measure a good question? With impurity — a quantity that is large when a box holds a mixture and drops toward zero when it holds one kind of thing. The most common choice, Gini impurity, looks like this.

$S$ is the set of rows you are looking at, and $p_k$ is the fraction of them belonging to class $k$. Read out loud: the probability that two draws from this box come out different. If everything in the box is the same, you always draw the same thing, so the value is 0; a fifty-fifty mixture gives 0.5. Think of it as a ruler for messiness.

The quality of a split is how much impurity the question removed.

$S_L$ and $S_R$ are the rows sent left and right by the question, $n_L$ and $n_R$ are their counts, and $n$ is the count before splitting. In words: the messiness before, minus the messiness after (averaged, weighted by how many rows landed on each side). Bigger means the question did more work.

What the learning algorithm does is blunt. Try every column crossed with every cut point, take the combination that maximizes Equation (2), and recurse on the children. That is the skeleton of CART, the classical method, and it is essentially what scikit-learn's decision tree does.

Two important properties fall out here. First, the search for a cut point happens independently per column, so income in dollars and tenure in years coexist without trouble — no standardization required. Second, the tree only ever looks at order, not magnitude, which makes it hard to upset with outliers. If one row has income typed in with an extra zero, the answer to "above $50,000?" does not change.

One tree is too clever, and breaks

The more questions you stack, the fewer rows survive to each leaf. Push it far enough and you get one row per leaf. At that point training accuracy is 100%, and the model is useless on anything new.

This is the textbook case of overfitting. A tree is expressive enough to memorize the accidental noise in the data as if it were a rule. Cap the depth and it swings the other way, too crude to catch what matters. A single tree is an awkward model: one knob, depth, and a tightrope between memorizing and not learning.

Ensembles: average them, or correct them

If one tree is unstable, build many and combine them. That is an ensemble, and there are two broad directions.

Bagging — build in parallel, then average. Resample the data at random each time, grow hundreds of deep trees, and take a vote. Individual trees flail, but if they flail in uncorrelated ways, averaging cancels it out. Random forests add "pick a random subset of columns at each split" on top, to stop the trees from resembling each other too closely. Easy and sturdy, but accuracy plateaus at merely decent.

Boosting — build in series, each one fixing the last. The idea is inverted. Grow one shallow, weak tree; then train the next tree on exactly what the first one got wrong; then a third on what is still wrong. Each tree can be so shallow it earns the name "stump". Chaining weak learners in series until they become strong — that is gradient boosting.

An easy way to hold the difference: bagging kills variance (solve the same problem repeatedly and average), boosting kills bias (make last round's mistakes into this round's homework).

Gradient boosting: chasing the residual

Now the mechanism in symbols. Write $F_m$ for the model after $m$ trees; the update is

where $f_m$ is the new tree and $\nu$ (nu) is the learning rate, a small number typically between 0.01 and 0.3. In words: blend a little of what the new tree has to say into what you already believed. Not blending all of it is the whole point — that restraint is what holds overfitting back.

So what does $f_m$ learn? This is where the "gradient" in the name comes from. For each row, differentiate the loss function with respect to the current prediction and flip the sign:

$L$ is the loss, $y_i$ the true label, $F(x_i)$ the prediction as it stands. This quantity says "which way, and how far, should this row's prediction move to reduce the loss" — and that is what the new tree is trained to predict. Under squared error it works out to exactly the residual (truth − prediction). So the intuitive story, "the next tree learns whatever is still wrong", is correct; the gradient is just its general form.

Which means gradient boosting is doing gradient descent in function space rather than parameter space. Instead of taking a step, you add a tree. The learning rate $\nu$ is literally the step size: too large and you overshoot, too small and you never arrive. It is the same relationship as ordinary gradient descent, with no asterisk.

What XGBoost, LightGBM and CatBoost actually add

The theory dates to the 1990s. What turned it into a commodity was the library work of the 2010s.

XGBoost (2016) evaluates splits using a second-order expansion of the loss. Alongside the first-order gradient $g_i$ it uses the second derivative $h_i$, which lets it solve for the leaf value analytically.

$I_j$ is the set of rows that fell into leaf $j$, and $\lambda$ is the regularization strength. Read it as "the total amount this leaf's rows want to move, divided by how easily they move" — and because $\lambda$ sits in the denominator, leaves holding few rows get pulled back toward zero. The overfitting defense is baked into the formula. XGBoost also learns, per split, which direction to send rows with a missing value (sparsity-aware split finding). Not having to impute is an unglamorous but real advantage in production.

LightGBM (2017) went after speed. It bins continuous values into a histogram up front to shrink the number of candidate cut points, and grows trees leaf-wise — always extending whichever leaf reduces loss most. For the same number of leaves that drives loss down faster than XGBoost's level-wise growth, at the cost of lopsided deep trees that overfit more readily, which is why num_leaves and min_data_in_leaf become the knobs that matter. Two more tricks feed the speed: sampling that preferentially keeps rows with large gradients (GOSS), and bundling sparse columns that are never non-zero at the same time (EFB).

CatBoost (2018) is built around categorical columns. Encoding a category by the mean of the target is powerful, but if the row's own label goes into that mean, the model is reading the answer. CatBoost imposes an ordering on the rows and computes each statistic from only the rows before it, closing that leak structurally rather than by convention.

In code

With a scikit-learn-compatible API the skeleton is short.

import lightgbm as lgb

model = lgb.LGBMClassifier(
    n_estimators=2000,      # number of trees (set high, let early stopping decide)
    learning_rate=0.05,     # the ν of Equation (3); smaller means more trees needed
    num_leaves=31,          # leaf-wise complexity — touch this first
    min_child_samples=20,   # minimum rows per leaf = the brake on overfitting
)
model.fit(
    X_train, y_train,
    eval_set=[(X_valid, y_valid)],
    callbacks=[lgb.early_stopping(100)],   # stop if 100 rounds bring no improvement
)

The part that matters is early_stopping. Boosting keeps driving training error down as long as you keep adding trees, so only validation data can tell you where to stop. Fixing the tree count by hand is the wrong instinct; set it generously and let the callback cut it off.

Why neural networks do not win on tables

There is no single reason, but the comparative studies (the benchmark by Grinsztajn and colleagues, the evaluation by Shwartz-Ziv and Armon) keep landing on the same points.

1. Tabular targets are not smooth. Relationships with a cliff in them — the decision flips at exactly $50,000 of income — are ordinary here. A split is a cliff, so a tree represents one in a single step, while a neural network, which prefers smooth functions, has to work to reproduce it.

2. Uninformative columns hurt. Real tables are full of irrelevant and duplicated columns. A tree picks only the column that helps at each split, so it ignores the rest by construction; a fully connected layer mixes every input together and drags the noise along.

3. Column identity is fixed. In an image, adjacency between pixels carries meaning. In a table, the column order means nothing, but each column itself has a specific meaning. The first layer of an MLP treats rotated inputs equivalently, which makes it poorly suited to exploiting that per-column identity.

On top of that, tabular datasets are often small — thousands to tens of thousands of rows — and the field never developed a culture of importing knowledge through pretraining. Recent work such as TabPFN, which reads a small table in as context, is changing that at the small end and is now a genuine option there. At production scale, though, GBDT remains the standard.

How this shows up on the job

Who touches it, and when. Credit scoring, fraud detection, churn prediction, demand forecasting, ad CTR, manufacturing yield analysis — whenever a table appears, a data scientist or ML engineer builds the baseline with GBDT first. "Run LightGBM before you try deep learning" is close to a professional norm. It also serves as a diagnostic: if the baseline is weak, suspect the problem framing or the features, not the model.

Parameters you touch first. For LightGBM: learning_rate, num_leaves, min_child_samples, feature_fraction, lambda_l2. For XGBoost: eta, max_depth, min_child_weight, subsample, colsample_bytree. The order is to drop the learning rate to around 0.05 and hand the tree count to early stopping, then tune complexity (num_leaves / max_depth), then regularization and sampling. Setting the number of trees by hand is a last resort.

Pitfalls that turn into incidents:

• No extrapolation. A tree predicts the average of the training rows in a leaf, so outside the range it saw during training the prediction flattens. Feed it revenue with an upward trend and it will under-predict the future forever. Detrending, differencing, or modelling the trend separately is mandatory — see time-series forecasting.
• Random splits on time series. Let future rows leak into training and validation looks unnaturally good. Always split on time.
• Target-encoding leakage. When you replace a category with the mean of the target, including the row's own label leaks the answer. Build it out-of-fold, or use something like CatBoost that prevents it by construction.
• One-hot encoding high-cardinality columns. One-hot an ID column with thousands of distinct values and the tree will happily manufacture splits that isolate a single row. Use the native categorical support in LightGBM or CatBoost instead.
• Reading feature_importances_ as causation. The default gain criterion is biased toward columns with many distinct values. When an explanation is required, look at per-row contributions with something like SHAP — and even then it is correlation, not cause.
• Cases that need monotonicity. "Income went up and the score went down" does not survive an underwriting review. XGBoost and LightGBM both offer monotone_constraints to force a column's effect to be monotonically increasing or decreasing.

How it comes up in interviews. "Why does GBDT do so well on tabular data?" "What is the difference between bagging and boosting?" "How do the learning rate and the number of trees relate?" — all three are answered by the sections above. For the last one, "halve the learning rate and you need roughly twice as many trees; it is step size versus number of steps" is enough.

Summary

• A decision tree keeps choosing good questions by how much impurity they remove. It shrugs off mismatched units and outliers, and its rules are readable.
• One tree is unstable, so you combine many: bagging averages the variance away, boosting hands each tree the previous one's mistakes to kill bias.
• Gradient boosting is gradient descent in function space. The learning rate is step size, the tree count is number of steps, and early stopping decides where to stop.
• Neural networks fail to win on tables because of three properties: cliff-shaped relationships, irrelevant columns, and per-column identity.

When a table shows up, start with LightGBM. Reaching for deep learning is fine — after you have failed to beat it.