Voting
Hard voting takes the majority label. Soft voting averages predicted probabilities and thresholds the mean, which is generally better because it uses confidence rather than discarding it — and which requires [calibrated](probability_calibration.html) probabilities to be meaningful.
Every member counts equally. That is the strength when the members are comparable and the weakness when they are not.
Look at the readout with the default settings. Three learners score roughly 62%, 66% and 90%, and the majority vote scores about 77% — worse than the best member alone. Two weak learners outvote a strong one whenever they agree, and they agree often enough to drag the result down.
This is not a contrived arrangement. It is what happens whenever an ensemble is assembled without checking whether the members are of comparable quality, and it is the reason "just ensemble it" is bad advice.
Stacking
Stacking replaces the fixed rule with a learned one. Train the base models, take their predictions as features, and fit a second model — the meta-learner — on those.
The meta-learner discovers what a vote cannot: which model to trust, and when. The learned weights are in the readout, and the strong learner attracts most of the weight while a weak one can go slightly negative — meaning the meta-learner has found that model to be worse than uninformative in some regions.
With the defaults, stacking matches the best single learner rather than beating it. That is an honest and common outcome, and it is still a gain over the vote: stacking recovered the best model automatically, where voting destroyed it.
Move the thresholds so that the three learners are closer in quality, and the picture changes — the vote catches up, because its assumption is finally true.
Getting stacking right
The essential detail is how the meta-learner's training data is produced.
Training the base models and then feeding their predictions on the same data to the meta-learner leaks badly. A model that overfits its training set produces suspiciously good predictions there, so the meta-learner learns to trust the most overfitted member.
The fix is cross-validated predictions: split the data into folds, and for each fold predict with base models trained on the other folds. Every prediction the meta-learner sees is then out-of-sample. This is what scikit-learn's StackingClassifier does, and it is why stacking costs roughly *k* times the training of its members.
A second convention: keep the meta-learner simple. Logistic regression is the standard choice. It has few base predictions to work with, and a flexible meta-learner overfits them readily.
When an ensemble is worth it
Members must be comparable in quality. Otherwise voting hurts, as above.
Members must make different mistakes. Three models that fail on the same rows combine into one model that fails on those rows. Checking the correlation of their errors is the diagnostic, and it is worth doing before building anything.
The gain must justify the cost. An ensemble multiplies training time, inference time, memory and the number of things that can break in production. A one-point gain on a leaderboard is worth it; a one-point gain in a service usually is not.
Where it goes wrong
Ensembling a strong model with weak ones by vote. Demonstrated above.
Fitting the meta-learner on in-sample predictions. It learns to trust the most overfitted base model.
A complex meta-learner. Few features, plenty of opportunity to overfit.
Assuming diversity. Two gradient-boosted models with different seeds are not diverse; check whether their errors actually differ.