The workhorses of tabular data.
WHAT A DECISION TREE DOES
Splits data repeatedly on feature values, predicting within each region.
WHAT IT PROVIDES
Interpretability, for shallow trees No need for scaling Native handling of non-linearity and interactions
WHAT IT SUFFERS FROM
Instability: small data changes produce different trees
Overfitting, when grown deep
WHAT RANDOM FORESTS DO
Train many trees on random subsets of data and features, and average them.
WHAT THAT PROVIDES
Much better generalisation, with little tuning.
WHAT GRADIENT BOOSTING DOES
Trains trees sequentially, each correcting the previous errors.
WHAT THAT PROVIDES
Typically the best performance available on tabular data.
WHAT IT REQUIRES
More careful tuning, and attention to overfitting.
WHAT THE MAIN IMPLEMENTATIONS ARE
Several mature libraries, differing in speed, categorical handling and defaults.
WHAT TO USE FOR TABULAR PROBLEMS
Gradient boosting, essentially always, as the strong approach.
WHY NOT DEEP LEARNING
It rarely beats boosting on tabular data, and costs far more.
WHAT TO TUNE FIRST
Learning rate, tree depth, and number of trees, with early stopping.
