Combining different models rather than many copies of one, and the case where a majority vote makes things worse.
Overview
Different from bagging and boosting
Bagging and boosting build many copies of one kind of model. Hundreds of trees — and combine them.
Voting and stacking combine models of *different* kinds: a logistic regression, a grad-boosted tree and a neural network, say. The hope is that models with different inductive biases fail on different rows. So Their errors partly cancel.
That hope is a rule, not a ensure, and this page is largely about the rule failing.
Stacking and Voting Ensembles
This module needs JavaScript: the
numbers are computed in the page rather than recorded.
Worth knowing
Voting combines predictions by majority, or by averaging probabilities. Every model counts equally.
Stacking trains a second model — the meta-learner — on the base models' predictions, so it can weight them.
Bagging and boosting combine many copies of one model. Voting and stacking combine different models.
An ensemble helps when the members make different mistakes. Three models that fail on the same rows gain nothing.
Stacking and Voting Ensembles
Mixing unlike models, when it helps, and a live demo of it hurting.
Voting
Hard voting takes the majority label. Soft voting means predicted chances and thresholds the mean. This Is often better because it uses trust rather than discarding it — and which needs calibrated chances to be real.
each member counts equally. That is the strength when the members are close and the weakness when they aren't.
Look at the readout with the default settings. Three learners score roughly 62%, 66% and 90%, and the majority vote scores about 77% — worse than the best member alone. Two weak learners outvote a strong one when they agree, and they agree often enough to drag the result down.
This isn't a contrived setup. It's what happens when an ensemble is assembled without checking whether the members are of close quality, and it is the reason "just ensemble it" is bad advice.
Stacking
Stacking replaces the fixed rule with a learned one. Train the base models, take their guesses as features, and fit a second model — the meta-learner — on those.
The meta-learner discovers what a vote can't: which model to trust, and when. The learned weights are in the readout, and the strong learner attracts most of the weight while a weak one can go slightly minus — meaning the meta-learner has found that model to be worse than uninformative in some regions.
With the defaults, stacking matches the best single learner rather than beating it. That is an honest and common outcome. It's still a gain over the vote: stacking got back the best model automatically. Here Voting destroyed it.
Move the thresholds so that the three learners are closer in quality, and the picture changes — the vote catches up, because its assumption is finally true.
Getting stacking right
The key detail is how the meta-learner's training data is made.
Training the base models and then feeding their guesses on the same data to the meta-learner leaks badly. A model that overfits its training set makes oddly good guesses there, so the meta-learner learns to trust the most overfitted member.
The fix is cross-validated guesses: split the data into folds, and for each fold predict with base models trained on the other folds. each guess the meta-learner sees is then out-of-sample. This is what scikit-learn's StackingClassifier does, and it's why stacking costs roughly *k* times the training of its members.
A second rule: keep the meta-learner simple. Logistic regression is the standard choice. It has few base guesses to work with, and a flexible meta-learner overfits them readily.
When an ensemble is worth it
Members must be close in quality. Or else voting hurts, as above.
Members must make different mistakes. Three models that fail on the same rows combine into one model that fails on those rows. Checking the link of their errors is the diagnostic, and it's worth doing before building any of it.
The gain must justify the cost. An ensemble times training time, inference time, memory and the number of things that can break in production. A one-point gain on a leaderboard is worth it. A one-point gain in a service often isn't.
When the majority is wrong, in numbers
Majority voting only helps when the members are close, and the math shows why. For three free classifiers the vote is correct when at least two are, P = Σ over the majority outcomes:
three members
majority-vote accuracy
0.62, 0.66, 0.90
0.83 — below the 0.90 member
0.80, 0.80, 0.80
0.90 — above any member
Three equal 80% models vote to 90%. This Is the ensemble win everyone hopes for: free errors partly cancel. But bolt a 90% model to two mediocre ones and the vote drops it to 83%. When the two weak members happen to agree they outvote the strong one, and two coin-ish models agree wrongly often enough to matter. That is voting's built-in assumption — equal competence — failing out loud, and it's why "just ensemble it" is bad advice without checking the members are close. Stacking fixes just this by learning the weights instead of fixing them: the meta-learner hands most of the weight to the strong member and can drive a weak one minus, getting back the best model where the vote destroyed it. (Real classifiers aren't free, so the true numbers shift with error link — which is itself the thing to measure before mixing any of it.)
Getting stacking right turns on one detail: the meta-learner must be trained on *out-of-sample* base guesses. Feed it guesses the base models made on their own training data and it learns to trust whichever member overfitted most, because an overfitted model looks oddly accurate there. The fix is cross-validated guesses — for each fold, predict with base models trained on the other folds — which is why stacking costs roughly k times the training of its members. Keep the meta-learner simple, too. Logistic regression is the standard choice, because it has few base guesses to work from and a flexible meta-learner overfits them readily.
Where it goes wrong with Stacking and Voting Ensembles
Ensembling a strong model with weak ones by vote. Shown above.
Fitting the meta-learner on in-sample guesses. It learns to trust the most overfitted base model.
A complex meta-learner. Few features, plenty of opportunity to overfit.
given range. Two grad-boosted models with different seeds aren't diverse. Check whether their errors really differ.
Questions people ask about Stacking and Voting Ensembles
What is the difference between voting and stacking? Voting combines predictions with a fixed rule, such as majority or average. Stacking trains a meta-model to learn how to combine them.
Hard or soft voting? Soft voting averages probabilities and usually does better when models produce reasonable probabilities. Hard voting counts class labels.
Why does stacking need cross-validated predictions? If the meta-model trains on predictions the base models made on their own training data, it learns to trust overfit outputs. Out-of-fold predictions avoid that.
Do ensembles always beat the best single model? No. If the models make the same mistakes, combining them adds little. The gain comes from diverse errors.
What should the meta-model be? Something simple, like logistic regression or ridge. A flexible meta-model overfits easily.
Is an ensemble worth the complexity? Sometimes for a small accuracy gain in competitions or high-stakes cases. In production, check that the gain justifies slower inference and harder maintenance.
Ensembling models that make different mistakes
Mixing models only helps when they're wrong about different rows. This measures that disagreement first, then shows what voting and stacking do with it.
example_01.pyscikit-learn
import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = make_classification(n_samples=4000, n_features=15, n_informative=6,
n_clusters_per_class=2, flip_y=0.12, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)
base = [
("logistic", make_pipeline(StandardScaler(), LogisticRegression(max_iter=3000))),
("tree", DecisionTreeClassifier(max_depth=8, random_state=0)),
("knn", make_pipeline(StandardScaler(), KNeighborsClassifier(15))),
("bayes", GaussianNB()),
]
preds = {}
for name, m in base:
preds[name] = m.fit(Xtr, ytr).predict(Xte)
print("%-10s accuracy %.4f" % (name, (preds[name] == yte).mean()))
print()
print("how often each PAIR gets a row wrong at the same time:")
names = list(preds)
wrong = {n: preds[n] != yte for n in names}
print("%-10s %s" % ("", "".join("%10s" % n for n in names)))
for a in names:
row = ""
for b in names:
both = (wrong[a] & wrong[b]).sum() / max(wrong[a].sum(), 1)
row += "%10.3f" % both
print("%-10s %s" % (a, row))
print(" (of rows THIS model got wrong, what fraction did THAT one also miss.)")
print()
any_right = (~np.all([wrong[n] for n in names], axis=0)).mean()
all_wrong = np.all([wrong[n] for n in names], axis=0).mean()
print(" at least one model is right on %.4f of rows." % any_right)
print(" all four are wrong together on %.4f." % all_wrong)
print(" that first number is the ceiling any combiner can reach.")
Output
The next block compares hard voting, soft voting and stacking on the same data.
example_02.pyscikit-learn
import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.ensemble import VotingClassifier, StackingClassifier
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = make_classification(n_samples=4000, n_features=15, n_informative=6,
n_clusters_per_class=2, flip_y=0.12, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)
base = [
("logistic", make_pipeline(StandardScaler(), LogisticRegression(max_iter=3000))),
("tree", DecisionTreeClassifier(max_depth=8, random_state=0)),
("knn", make_pipeline(StandardScaler(), KNeighborsClassifier(15))),
("bayes", GaussianNB()),
]
names = [n for n, _ in base]
hard = VotingClassifier(base, voting="hard")
soft = VotingClassifier(base, voting="soft")
stack = StackingClassifier(base,
final_estimator=LogisticRegression(max_iter=3000), cv=5)
for name, m in (("hard voting (majority)", hard),
("soft voting (avg probability)", soft),
("stacking (model on outputs)", stack)):
print("%-34s cv %.4f" % (name, cross_val_score(m, X, y, cv=5).mean()))
print("%-34s cv %.4f"
% ("best single model",
max(cross_val_score(m, X, y, cv=5).mean() for _, m in base)))
print()
stack.fit(Xtr, ytr)
print("what stacking learned about which model to trust:")
for name, w in zip(names, stack.final_estimator_.coef_[0]):
print(" %-10s weight %+7.4f" % (name, w))
print(" a negative weight means the combiner leans against that model.")
print()
print("soft beats hard because 0.51 and 0.99 carry different evidence.")
print("stacking needs cv predictions or training optimism leaks in.")
Output
Check yourself
0 of 3
Answer without scrolling back up.
Why can a majority vote score worse than its best member?
It is what happens whenever an ensemble is assembled without checking the members are comparable in quality - which is why 'just ensemble it' is bad advice.
Why must the meta-learner be trained on cross-validated predictions?
Every prediction the meta-learner sees has to be out-of-sample. It is why stacking costs roughly k times the training of its members.
What condition must hold for an ensemble to help at all?
Three models that fail on the same rows combine into one model that fails on those rows. Checking the correlation of their errors is the diagnostic to run first.
Cheat sheet
Stacking and Voting Ensembles
Combine trees, linear models and neighbours into one ensemble. Compare hard voting, soft voting and stacking on the same scored data.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.