Logistic Regression vs Random Forest vs Gradient Boosting: When the Simple Model Wins

It was not. Here are the real results, from code that was actually executed, and what they teach about choosing a model.

The dataset

A customer churn table with four features: tenure, monthly charges, support calls and contract type. Roughly one customer in five has churned. Every model below is trained on the same training set and scored on the same sealed test set.

Step 1: a single decision tree

A tree restricted to two levels of questions already scores 0.810 on the test set. Remove the depth limit and it grows to depth 14, reaches a perfect 1.000 on the training data, and scores exactly the same 0.810 on the test set. Every extra question was memorising the training data, not learning anything new: textbook overfitting.

# OUTPUT
# depth=2     train acc=0.832  test acc=0.810
# unlimited   train acc=1.000  test acc=0.810
# unlimited tree depth reached: 14

Step 2: a random forest

Two hundred trees, each trained on a random sample of rows and features, voting together. Accuracy moves by only two points, but AUC, which measures how well the model ranks churners above loyal customers, jumps from 0.731 to 0.863. Accuracy alone would have hidden most of that improvement.

from sklearn.ensemble import RandomForestClassifier

forest = RandomForestClassifier(n_estimators=200, random_state=0)
forest.fit(X_train, y_train)

# OUTPUT
# single tree (unlimited) test acc : 0.810   AUC : 0.731
# forest (200 trees)      test acc : 0.830   AUC : 0.863

Step 3: gradient boosting, against both rivals

from sklearn.ensemble import GradientBoostingClassifier

gb = GradientBoostingClassifier(
    n_estimators=200, learning_rate=0.05, max_depth=3, random_state=0
)
gb.fit(X_train, y_train)
gb.score(X_test, y_test)

# OUTPUT
# logistic regression   test acc=0.870
# random forest         test acc=0.830
# gradient boosting     test acc=0.840

Plain logistic regression beat both tree ensembles. This is not a bug, and it is not unusual. Churn in this dataset is driven by a mostly linear combination of tenure, support calls and contract type, exactly the kind of relationship a linear model handles well. The extra flexibility of trees bought nothing.

Boosting also showed diminishing returns quickly. Its first tree alone reached 78.0% accuracy; by around eighty trees it had essentially arrived at its final score, and the remaining hundred and twenty trees added almost nothing.

The plot twist: a better feature beat a better algorithm

Adding one engineered column, support calls divided by tenure, to the unchanged boosting model moved accuracy from 0.840 to 0.855. That is a larger gain than the gap between several of the algorithms above.

df["calls_per_month_tenure"] = df["support_calls"] / (df["tenure"] + 1)
# refit the SAME boosting model, changing nothing except this one column

# OUTPUT
# without engineered feature : test acc=0.840  AUC=0.886
# with engineered feature    : test acc=0.855  AUC=0.888

How to choose

Model	Reach for it when
Logistic regression	The relationship is roughly linear, or you need a fast, interpretable baseline
Random forest	You want strong performance with minimal tuning and some resistance to overfitting
Gradient boosting	You have time to tune carefully and want to squeeze out the last few points

The rule that matters more than the table: there is no “best” algorithm independent of the data. Always train a simple baseline first, compare honestly on a sealed test set, and check more than one metric. If you are new to the distinction between the model families themselves, start with supervised vs unsupervised learning.

Key takeaways

• On this churn data: logistic regression 0.870, gradient boosting 0.840, random forest 0.830.
• An unlimited decision tree hit 1.000 on training data and gained nothing on the test set.
• AUC revealed an improvement (0.731 to 0.863) that accuracy mostly hid.
• One good engineered feature moved the needle more than switching algorithms.

---

Learn this properly, with code that actually ran

These results come from Part 4 of the Machine Learning Starter Kit (Chapters 14 to 17), a 172-page handbook where every listing was executed and every output is the real one. Just starting out? Get the free 90-Day Machine Learning Roadmap first.