Drag one point around and watch a least-squares line follow it. Twenty-four rows say one thing; the twenty-fifth overrules them.
Overview
Drag it
The chart holds twenty-four points along a clear upward line, plus one highlighted point you can move. The dashed line is the least-squares fit without it. The solid line is the fit with it.
Drag the point straight up. The solid line rotates. Drag it to the far right and up, and the line follows it much further. The readout reports both slopes and the percentage difference between them, and it doesn't take much to change the answer by tens of per cent.
Twenty-four rows are saying one thing. The twenty-fifth overrules them.
Outliers and Influence
This module needs JavaScript: the
numbers are computed in the page rather than recorded.
Worth knowing
Least squares minimises squared error, so a point twice as far off pulls four times as hard.
An outlier far along the x axis has high leverage: it sits at the end of the lever and moves the line most.
A point can be extreme and correct. Deleting it because it is inconvenient is how findings stop being true.
Trees, medians and quantile regression are robust to outliers. Means, least squares and standard deviations are not.
Outliers and Influence
One row out of twenty-five, and the reason it can decide your answer.
Why squaring does this
Least squares chooses the line cutting the sum of squared residuals. Squaring is what makes a single point so powerful: a point twice as far from the line adds four times the fine, and one ten times as far adds a hundred times.
The fit is so willing to move a long way to reduce one large residual, even at the cost of small grows across every other point. That isn't a bug in the setup. It's what the goal function asks for.
Hat values: where it sits matters as much as how far off it is
Drag the point up while keeping it near the middle of the x range, then drag it up while keeping it at the far right. The second spot moves the line much more.
That difference is the hat value. A regression line pivots roughly about the mean of x, so a point near that mean is close to the pivot and has little turning effect no matter how far off it is vertically. A point at the extreme of x sits at the end of a long lever.
The gap is worth having in three words:
Outlier — unusual in y, a large residual.
High hat value — unusual in x, far from the pivot.
Influential — both, so dropping it changes the model. This is the one that matters, and Cook's distance is the standard way to measure it.
A point with a high hat value that happens to lie on the trend is harmless. A moderate outlier in the middle is harmless. The combination isn't.
Before you remove anything
The temptation is to delete the point. Resist it long enough to ask which of three things it's.
An error. A misplaced decimal, a sensor fault, a unit mix-up, a placeholder like 999 or −1 meaning "unknown". Fix it or drop it, and record that you did.
A different group. The row is correct but doesn't belong to the question — a wholesale order in a table of retail purchases. Exclude it and say so, or model the groups apart.
A real, extreme member of the group. The row is correct and belongs. This one you keep. Dropping genuine extremes because they're inconvenient is how a result stops showing the world, and it's much more common than deliberate fraud.
If you can't tell which, report the analysis both ways. A conclusion that lives the point and a conclusion that depends on it are different findings, and the reader is entitled to know which they've.
Ways to be less fragile
Solid loss. Huber loss is squared for small residuals and linear for large ones, so a distant point pulls hard but not as n² hard. RANSAC fits on random subsets and keeps the consensus.
Quantile regression. Models the median rather than the mean, and the median doesn't move when one point is dragged to infinity.
Winsorising. Clip values at, say, the 1st and 99th percentiles. Keeps the row, caps its sway.
Use a model that doesn't care. Trees split on order, not on distance, so a value of a million and a value of a thousand are the same split if nothing lies between them.
Transform. A log transform on a skewed plus column often turns "outliers" into plain members of a normal-shaped spread, which is often a sign the column was on the wrong scale.
Why position matters as much as distance
The reason one far-off point wrecks a fit and another leaves it alone is the hat value, and squared error is what makes it bite. Least squares cuts the *sum of squared* residuals, so a point twice as far off adds four times the fine and one ten times as far adds a hundred times. The line will travel a long way to reduce a single large residual. But a regression line pivots roughly about the mean of x. So How far a point sits along x decides how much turning force its vertical error applies:
the odd point
effect on the slope
large residual, near the middle of x
small — close to the pivot
large residual, at the extreme of x
large — the end of a long lever
That gives three distinct things worth separating: an outlier is unusual in y, a high hat value is unusual in x, and only their combination — influential — really moves the model. This is what Cook's distance measures. A point with a high hat value sitting on the trend is harmless. A mid-range outlier is harmless. The far-right outlier is the one that quietly rewrites the answer. Which is also why "delete any of it beyond three standard deviations" is self-defeating: the outlier has already inflated the standard deviation the rule depends on.
Where it goes wrong with Outliers and Influence
Deleting by rule. "Beyond three standard deviations" uses the standard deviation, which the outlier itself has already inflated.
Treating placeholders as data. 999 for unknown age, −1 for missing. These aren't outliers. They're missing values wearing a number.
Dropping points until the model fits. At that stage the model shows the pick, not the effect.
Checking only y. Points with high hat values can be plain in y and still dominate the fit.
Questions people ask about Outliers and Influence
Is every outlier a problem? No. An outlier is an unusual value, and it matters only if it changes the fit. A point that sits on the trend line is harmless.
What is the difference between an outlier and an influential point? An outlier has an unusual response given its inputs, while an influential point changes the fitted model when removed. Points far from the other inputs are the usual cause of influence.
How do I measure influence? Cook's distance combines residual size and distance from the other inputs in one number. A common rule of thumb flags values above 4/n for a closer look.
Should I delete outliers? Only after checking whether they are errors. If the value is real, removing it hides something true about your data, so consider a different model or transform instead.
What can I do besides removing them? Use a loss that is less sensitive to large errors, such as Huber or quantile regression. Or transform the target, or use median-based statistics.
Why does squared error make outliers worse? Squaring makes one large residual count far more than many small ones, so the fit bends towards the extreme point.
An outlier and an influential point are not the same thing
One far-out row can leave a fit untouched while another moves it fully. The difference is the hat value, and it is computable.
example_01.pyscikit-learn
import numpy as np
from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(0)
x = np.linspace(0, 10, 60)
y = 2.0 * x + 5.0 + rng.normal(0, 1.5, 60)
X = x.reshape(-1, 1)
def fit(Xa, ya, label):
m = LinearRegression().fit(Xa, ya)
print(" %-34s slope %7.4f intercept %7.4f" % (label, m.coef_[0], m.intercept_))
return m
print("the true relationship is y = 2x + 5.")
base = fit(X, y, "clean data")
xm, ym = 5.0, 60.0
Xa = np.vstack([X, [[xm]]]); ya = np.r_[y, ym]
mid = fit(Xa, ya, "one wild point at x=5 (middle)")
xe, ye = 10.0, 65.0
Xb = np.vstack([X, [[xe]]]); yb = np.r_[y, ye]
edge = fit(Xb, yb, "one wild point at x=10 (edge)")
print()
print("both sit ~%.0f above the true line." % (ym - (2 * xm + 5)))
print(" x=5 moved the slope by %.4f" % abs(mid.coef_[0] - base.coef_[0]))
print(" x=10 moved the slope by %.4f" % abs(edge.coef_[0] - base.coef_[0]))
print("x=5 is the mean of x, so that point has no lever at all: it pushed")
print("the whole line up without tilting it. the edge point tilted it.")
Output
The next block computes hat values and Cook's distance to show how influence is measured.
example_02.pyscikit-learn
import numpy as np
from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(0)
x = np.linspace(0, 10, 60)
y = 2.0 * x + 5.0 + rng.normal(0, 1.5, 60)
X = x.reshape(-1, 1)
Xa = np.vstack([X, [[5.0]]]); ya = np.r_[y, 60.0]
Xb = np.vstack([X, [[10.0]]]); yb = np.r_[y, 65.0]
def leverage(Xc):
Z = np.column_stack([np.ones(len(Xc)), Xc])
H = Z @ np.linalg.solve(Z.T @ Z, Z.T)
return np.diag(H)
print("leverage depends only on x, never on y:")
for label, Xc in (("point at x=5 ", Xa), ("point at x=10", Xb)):
h = leverage(Xc)
print(" %s leverage %.4f (average is %.4f)" % (label, h[-1], h.mean()))
print(" far from the mean of x = long lever.")
print()
print("Cook's distance combines both: how far off the line, times leverage.")
def cooks(Xc, yc):
m = LinearRegression().fit(Xc, yc)
resid = yc - m.predict(Xc)
h = leverage(Xc)
mse = (resid ** 2).sum() / (len(yc) - 2)
return resid ** 2 * h / (2 * mse * (1 - h) ** 2)
for label, Xc, yc in (("added at x=5 ", Xa, ya), ("added at x=10", Xb, yb)):
d = cooks(Xc, yc)
print(" %s Cook's D %.4f (investigate above %.4f)" % (label, d[-1], 4/len(yc)))
print()
print(" outlier -- unusual y for its x. may cost you nothing.")
print(" high leverage -- unusual x. dangerous whether or not y is odd.")
print(" influential -- actually moved the fit. Cook's D measures this.")
Output
The next block shows how resistant estimators handle the same outlier.
example_03.pyscikit-learn
import numpy as np
from sklearn.linear_model import LinearRegression, HuberRegressor, RANSACRegressor
rng = np.random.default_rng(0)
x = np.linspace(0, 10, 60)
y = 2.0 * x + 5.0 + rng.normal(0, 1.5, 60)
X = x.reshape(-1, 1)
Xb = np.vstack([X, [[10.0]]]); yb = np.r_[y, 65.0]
print("robust fits, on the edge-outlier data:")
edge = LinearRegression().fit(Xb, yb)
print(" ordinary least squares slope %7.4f" % edge.coef_[0])
print(" Huber slope %7.4f"
% HuberRegressor().fit(Xb, yb).coef_[0])
print(" RANSAC slope %7.4f"
% RANSACRegressor(random_state=0).fit(Xb, yb).estimator_.coef_[0])
print(" true slope 2.0000")
print()
print("Huber switches from squared to absolute error past a threshold,")
print("so a far-out point stops getting quadratically more say. RANSAC")
print("fits on random subsets and keeps the fit most points agree with.")
print()
print("deleting the point is a decision about what your data means.")
print("a sensor glitch, delete it. a genuine extreme customer, deleting")
print("means the model no longer covers them.")
Output
Check yourself
0 of 3
Answer without scrolling back up.
Why does least squares react so strongly to a single distant point?
The objective is willing to move a long way to reduce one large residual, even at a small cost to every other point. That is what the loss function asks for.
What makes a point 'influential' rather than merely an outlier?
The line pivots about the mean of x, so a point far along x sits at the end of a long lever. Extreme in y near the pivot is harmless; the combination is not.
What is wrong with removing anything beyond three standard deviations?
The rule uses a statistic the outlier corrupts, so a large enough outlier widens the threshold enough to protect itself. Robust measures like the median absolute deviation avoid this.
Cheat sheet
Outliers and Influence
Move one data point and watch the regression line follow it. Use Cook's distance and hat values to find the rows that control your model.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.