The baseline worth beating.
WHAT LINEAR REGRESSION DOES
Fits a weighted sum of features to predict a number.
WHAT LOGISTIC REGRESSION DOES
Fits a weighted sum, passed through a function producing a probability.
WHY THEY REMAIN IMPORTANT
Fast to train and to run Fully interpretable Strong baselines Well understood statistically
WHAT THE COEFFICIENTS MEAN
The effect of a unit change in a feature, holding others constant.
WHY THAT INTERPRETATION REQUIRES CARE
Correlated features make individual coefficients unstable and misleading.
WHAT REGULARISED VARIANTS PROVIDE
Stability with many features, and automatic feature selection.
WHAT THEY REQUIRE OF FEATURES
Scaling, since penalties apply to coefficient magnitude Encoding of categories Explicit interaction terms, where interactions matter
WHY THAT LAST POINT
Linear models cannot discover interactions themselves.
WHAT THEY HANDLE POORLY
Non-linear relationships, unless transformed Complex interactions
WHERE THEY ARE STILL PREFERRED
Regulated settings requiring explanation Very high-dimensional sparse data Situations demanding extreme speed
WHAT TO ALWAYS DO
Fit one first, and record its performance.
