Classical ML is not dead: why gradient-boosted trees still beat deep learning on tabular data

Most of the data that decides whether a business works is tabular: invoices, claims, transactions, sensor readings, building elements, support tickets, patient records. And on that data — the plain two-dimensional tables nobody writes conference papers about — gradient-boosted decision trees still win the majority of realistic tasks. That is not nostalgia and it is not a lack of imagination. It is a structural property of how trees and neural networks carve up a feature space, and it has been measured repeatedly by independent groups on independent benchmarks.

A serious engineer does not choose a model family because it is fashionable. They choose it because they can explain why it wins. So this article does three things: it states the evidence without hand-waving, it explains the mechanism behind the result, and then it draws the boundary precisely — here is where the tree still wins, here is where deep learning genuinely takes over, and here is how to decide for your own table in an afternoon.

The claim, stated plainly

On medium-sized tabular datasets — call it hundreds to tens of thousands of rows, with a mix of numeric and categorical columns — boosted trees (XGBoost, LightGBM, CatBoost) are the strongest default. Not the most exciting default. The strongest one. And this position is defensible because three separate lines of published work arrived at it from different directions.

Grinsztajn, Oyallon and Varoquaux — "Why do tree-based models still outperform deep learning on typical tabular data?" (NeurIPS 2022, Datasets and Benchmarks Track, arXiv:2207.08815). The authors benchmarked a broad set of tree ensembles against a broad set of neural architectures across dozens of tabular datasets and concluded that tree-based models remain the best approach on typical tabular data — and, crucially, that the gap is not an artefact of tuning effort. They then went further than most benchmark papers and offered hypotheses for why.
Shwartz-Ziv and Armon — "Tabular Data: Deep Learning is Not All You Need" (arXiv:2106.03253). Their comparison found XGBoost performing better than the deep tabular models in most cases even without hyperparameter tuning, and additionally that ensembles combining the two families can beat either alone. Both halves of that finding matter, and I come back to the second one at the end.
Gorishniy, Rubachev, Khrulkov and Babenko — "Revisiting Deep Learning Models for Tabular Data" (NeurIPS 2021, arXiv:2106.11959). This one cuts the other way in part, and that is exactly why it is worth reading: carefully designed, properly preprocessed simple architectures turned out to be competitive with specialised ones. When a plain MLP with the right preprocessing matches an exotic architecture, the exotic architecture was never the source of value — the preprocessing and the design discipline were.

Then add the survey literature. Borisov et al., "Deep Neural Networks and Tabular Data: A Survey" (arXiv:2110.01889) reviews roughly a decade of work in this space, and its conclusion is the nuanced one that most arguments skip: gradient-boosted trees dominate on medium-sized data, while deep networks show genuine advantages on large datasets and in multimodal settings. That nuance is this article. The headline is "trees win", the useful part is the boundary.

"Why do tree-based models still outperform deep learning on typical tabular data?" — the title of Grinsztajn, Oyallon and Varoquaux, NeurIPS 2022. Years later, the best answer the field has is still inductive bias, not hyperparameter search.

Why: the smoothness bias

A decision tree splits on one feature at a time. Every split is a threshold on a single column, so the model's decision region is a set of axis-aligned boxes in feature space. That geometry matches how tabular data is actually produced. A rule like "deny the claim if the amount exceeds this band and the policy was opened less than ninety days ago" is two thresholds on two columns. Trees represent it with two splits. This is not a coincidence — business rules and physical constraints are written as thresholds on individual variables, because that is how humans reason about them.

A multilayer perceptron does something fundamentally different. It learns linear combinations of all features at once and then applies nonlinearities to those combinations. That makes the network remarkably good at smooth, shared, distributed structure — and it makes it rotationally invariant: a linear combination of an informative feature and an uninformative feature is still a perfectly usable input for the first layer. The cost is that an axis-aligned boundary which a tree expresses with three splits may require many layers and far more parameters for a network to approximate, because the network has to construct the alignment first.

Grinsztajn and co-authors frame this as an inductive-bias mismatch, and the consequence is measurable: neural networks are biased toward smooth solutions, and tabular targets frequently are not smooth. The textbook illustration is the XOR-style target, where the label depends on a threshold of one variable combined with a threshold of another. A tree nails it with a handful of splits. An MLP needs substantially more capacity to represent the same function, because the true boundary has nothing to do with any linear combination of the inputs.

This is not a failure of deep learning. It is deep learning doing exactly what its architecture was designed to do — exploit smoothness and shared structure — in a setting where that design assumption partially fails to hold. Reading it as "neural nets are bad at tables" misses the point and produces the wrong decision. The correct reading is: the assumption is what you are buying, and here it does not pay.

Why: uninformative features and heterogeneous columns

Two further mechanisms from the same body of work matter more in day-to-day practice than the smoothness argument, because they are the ones you can feel immediately.

Uninformative features. Trees implicitly select: a column that carries no signal simply never gets chosen for a split, and adding fifty useless columns leaves the chosen structure largely intact. Neural networks are not as lucky. Every input feature enters the first layer, so the network has to learn from gradient signal to suppress the ones that do not matter — a task that costs capacity and training time. The Grinsztajn benchmark reports specifically that adding uninformative features damages neural-network performance more than tree performance, which is precisely what you would expect if one family has built-in feature selection and the other does not.
Heterogeneous, mixed-type columns. A real table is a numeric column, a categorical column, a datetime, and a free-text field in the same row. Trees accept that mixture almost natively. A decision tree is invariant to monotone transformations of a single feature, so the scale of a numeric column does not change the set of possible splits. Neural networks require the designer to handle each type explicitly: normalise numerics, one-hot or embed categoricals, encode datetimes into cyclical or ordinal features, decide what to do with missingness. Each of those is a place to introduce a mistake and an extra knob to tune. The consequence is not that networks cannot handle mixed tables — they can — but that the tree gets there with far fewer design decisions, and design decisions are where projects actually lose time.

The operational reasons that never make it into the abstract

The modelling argument is only half of the story. The other half is that a model which is marginally more accurate but harder to run is frequently the wrong model to ship.

Training time and hardware. Boosted trees train on CPU with well-optimised histogram splitting, and they train fast. A neural network usually wants a GPU to be practical at any real scale; a tree ensemble is happy on the same machine that runs your ETL. For a model that gets retrained nightly, or on every data refresh, that difference outranks a fraction of a point in AUC.
Tuning effort. Gradient boosting with sane defaults is remarkably good out of the box. Shwartz-Ziv and Armon explicitly noted that XGBoost beat the deep models even without hyperparameter tuning. A neural model, by contrast, arrives with a learning rate, a schedule, a batch size, a width, a depth, a dropout rate, and an optimiser — and its performance is sensitive to all of them.
Missing values and outliers. Modern GBDT implementations handle missing values natively and are robust to outliers because their splits are rank-based thresholds. An MLP needs imputation, and imputation is a modelling choice you then have to defend to whoever owns the data.
Interpretability and governance. Feature importance, SHAP values, partial dependence, and — most important of all — a human-readable trace of which features drove a specific prediction. If your model has to be explained to a risk team, a regulator, or a client, the tree hands you the explanation. That is not a soft benefit. In regulated domains it decides which model is permitted to ship.
Deployment surface. A boosted ensemble compiles to a small artifact and scores in a few milliseconds on a single core, which means it can live inside the application or inside the analytics engine rather than behind a dedicated inference service. Fewer moving parts, fewer failure modes, one less thing to page someone about at 3am.

The exceptions: where deep learning genuinely wins

If trees always won, this would not be an engineering decision — it would be a rule. They do not. The published literature and my own experience agree closely on where the boundary runs, and it runs in five places.

Large data. The survey literature is consistent that as dataset size grows into the hundreds of thousands or millions of rows, the deep-learning advantage appears and then widens. Trees have finite capacity to exploit new data once the effective number of distinct decision paths saturates; a network with enough parameters keeps improving with more examples, provided you can pay for the compute. If your table is large and will keep growing, the calculus genuinely changes.
Multimodal inputs. If the row is not only numbers and categories but also an image, an audio clip, a block of text, or a graph neighbourhood, a neural network is the only serious answer, because it can learn a shared representation across modalities. Borisov et al. call this out explicitly. A tree cannot embed a paragraph; a transformer can, and then the tabular columns join the same representation and benefit from it.
High-cardinality categoricals. Embeddings for a category with tens of thousands of levels — user IDs, SKUs, postcodes, equipment tags — are a neural-network strength that trees handle awkwardly. If your table is mostly a sparse identity space, an embedding layer is the right tool and a one-hot tree split is the wrong one.
Sequences and repeated structure. A table where each entity has many timestamped rows is not really a table; it is a sequence problem wearing a table's clothes. Sequence and attention architectures exploit that structure directly. Grinsztajn and co-authors specifically flag the case where rows are far from independent, and that is where a neural approach earns its keep.
Multi-task learning and shared representations. Training one model to predict many related targets, or to act as a shared encoder for several downstream heads, is a neural pattern. Trees do one target at a time, and stacking fifteen of them to serve fifteen heads is not the same thing as learning a shared representation.

The new twist: tabular foundation models

The most interesting recent development is that the deep-learning side largely stopped trying to beat the tree with a better architecture and attacked the problem from a different direction: pre-training. TabPFN (Hollmann et al., arXiv:2207.01848) is a transformer pre-trained on synthetic tabular tasks that then classifies a new small table in a single forward pass — with no task-specific training at all. The authors positioned it for small tables, on the order of a thousand rows, and the v2 line extended the approach to larger and more realistic tables; that successor work was published in Nature in 2025.

This is the right kind of disruption, because it attacks the actual weakness of neural methods on tabular data. They were always data-hungry and labour-intensive to tune. A pre-trained prior removes both the hunger and most of the tuning: you hand it the table and it answers. If tabular foundation models keep improving, they are the thing that will displace the "trees win" default — not yet, but that is the direction of travel. The honest reading in 2026 is that gradient boosting is still the incumbent default and that the foundation-model line is the credible challenger. Anyone claiming the default has already flipped is reading announcements instead of benchmarks.

A decision framework you can run in an afternoon

Here is the sequence I actually use. It takes half a day, not a quarter, and it is deliberately biased toward shipping something.

Build the baseline first, and make it a real one. LightGBM or CatBoost with sensible defaults, a proper train/validation split that respects time if the data is temporal, and a metric that matches the business decision rather than a metric that is easy to compute. Do not skip this step. The baseline is the number every later effort has to beat, and in my experience it is far harder to beat than people expect.
Fix the data before you fix the model. Row count, feature count, missingness per column, cardinality per categorical, and the time structure of the split. A large share of "deep learning is better" results in practice turn out to be "we cleaned the data carefully while we were building the neural model". Make the cleaning part of both experiments, or the comparison is meaningless.
Ask the four boundary questions. Is the table large and growing into the hundreds of thousands of rows? Are there non-tabular inputs in the same pipeline? Are the categoricals high-cardinality enough to want embeddings? Is the row actually a sequence for an entity? Any "yes" moves you toward neural methods.
If all four answers are no, stop. Ship the tree, spend the remaining time on feature engineering and on the data pipeline, and leave the architecture debate for a project that needs it. In tabular work, feature engineering still moves performance more than architecture does — the least glamorous and highest-return part of the job.
If one or more answers are yes, build the neural model and compare it against the baseline — not against the previous neural model. Then, before shipping, run the operational checklist: training time per refresh, artifact size, latency budget, and whether you can explain an individual prediction to the person who has to defend it.
Consider the ensemble. Shwartz-Ziv and Armon found that combining the two families can improve on either alone. When the tree is half a point behind and the network is half a point ahead, a stacked or averaged model can be the real winner — at the cost of running and monitoring two systems. That trade is sometimes correct and always worth stating out loud.

What this looks like on a real stack

On the kmail.at stack, tabular work rarely starts in a notebook. It starts in the query engine. The analytics layer already runs DuckDB over Parquet columns, which means training features and inference-time features can come from the same SQL. That single-source discipline removes the most common cause of a good tree model underperforming in production: train/serve skew. The model was fine; the features it saw at inference time were built by a different code path.

Building data follows the same logic. A BIM model is not a drawing; it is a structured record of elements with properties, and once that record lands in a table — element type, level, material, area, cost, maintenance interval — the prediction tasks are classical ones: cost regression, defect classification, anomaly detection on operational readings. Those are boosted-tree problems with explainability requirements attached, because a facility manager asking "why was this flagged?" needs an answer, not an embedding vector. The same holds for the agentic systems: when an agent needs a cheap, explainable, deterministic judgement call over structured facts — is this claim anomalous, is this record inconsistent, does this specification contradict that schedule — a tree is often the right tool behind the tool, with the language model handling the reasoning and the interface.

The pattern that matters is not "use trees". It is that the model sits at the end of a pipeline, and the pipeline around it — feature definitions, retraining cadence, monitoring, explanation surface — decides whether the model is useful. A marginally worse model in a well-instrumented pipeline beats a better model in a fragile one, every time.

How to benchmark honestly

The tabular-vs-deep literature is easy to read selectively in either direction, so if you run the comparison yourself, hold yourself to a few rules.

Same folds, same preprocessing budget. If the neural model gets a tuning sweep and the tree gets library defaults, you measured your effort, not your architectures. Give both a budget and report both.
Report the metric that decides, not the metric that is easy. Accuracy on an imbalanced table is theatre. Use the metric that maps to money, risk, or time saved, and report the confusion structure behind it.
Report variance across seeds. Tree ensembles are fairly stable; deep models are less so. A single-seed gap of a fraction of a percent is noise, and shipping on noise is how teams end up maintaining a neural model that never actually paid for itself.
Time the full loop. Training time, inference latency, and retraining cadence. A model that takes four hours to retrain changes your release process; a model that takes four minutes does not.
Count the design decisions. This is the least quantified and usually the most decisive variable in which model actually ships. Every knob is a future bug and a future maintenance cost.

The bottom line

Gradient-boosted trees are the correct default for tabular data in 2026, and the reason is inductive bias, not inertia: axis-aligned splits, implicit feature selection, native handling of mixed types and missing values, a CPU-friendly training loop, and an explanation surface you can hand to a reviewer. Deep learning takes over exactly where its assumptions start to hold: large tables, multimodal rows, high-cardinality embeddings, sequences, and shared multi-task representations. Tabular foundation models are the credible challenger to that default, and they are not yet the default.

So the real decision is not "classical or modern". It is whether your data is shaped like a spreadsheet or shaped like a sequence or a perception problem. Spreadsheets get boosted trees. Everything else gets a neural network, a foundation model, or both. Get that call right and most of the architecture debate disappears — which is the point. The architecture was never the hard part; the shape of the data was.

Pick the model that matches the shape of your data, not the shape of the current hype cycle. On tables, that answer has been stable for years — and the interesting part is knowing exactly when it stops being true.

Sources

Grinsztajn, Oyallon, Varoquaux (2022): "Why do tree-based models still outperform deep learning on typical tabular data?" — NeurIPS 2022 Datasets and Benchmarks Track — https://arxiv.org/abs/2207.08815
Shwartz-Ziv, Armon: "Tabular Data: Deep Learning is Not All You Need" — https://arxiv.org/abs/2106.03253
Gorishniy, Rubachev, Khrulkov, Babenko (2021): "Revisiting Deep Learning Models for Tabular Data" — NeurIPS 2021 — https://arxiv.org/abs/2106.11959
Borisov, Leemann, Seßler, Haug, Pawelczyk, Kasneci: "Deep Neural Networks and Tabular Data: A Survey" — https://arxiv.org/abs/2110.01889
Hollmann, Müller, Eggensperger, Hutter: "TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second" — https://arxiv.org/abs/2207.01848
TabPFN v2: Hollmann et al., published in Nature (2025) — the successor line extending the prior-fitted approach to larger, more realistic tables.
Kaggle State of Machine Learning and Data Science survey: the industry-side corroboration that gradient-boosting implementations dominate tabular work — https://www.kaggle.com/kaggle-survey-2021

Where this article states a number, it is a scope or a claim taken from the cited paper (for example TabPFN's small-table regime). Where it describes practice — tuning effort, deployment cost, the shape of a real pipeline — it is labelled as practice and drawn from hands-on work rather than from a measurement.