Fit a line to a dataset by minimizing the Mean Squared Error (MSE). Drag the regression line manually to see how the residuals (red lines) change, or compute the exact best fit.
Overview
Overview
Linear Regression is a fundamental algorithm in machine learning used to model the relationship between two variables. The goal is simple: find the straight line that best fits a set of data points. This line, often called the "regression line" or "hypothesis," is represented by the classic equation y = mx + c, where 'm' is the slope and 'c' is the y-intercept.
Controls
Slope (m)
Intercept (c)
Current Loss0.00
Optimal Loss0.00
Distance to Optimal0.00
Tip: You can click and drag individual blue data points, or drag the line handles to manually adjust the fit. Drag the background to pan the view.
Visualization
Drag to pan & edit points
Loss Function
Mean Squared Error
MSE =
1n
∑ (yi - ŷi)²
Current Hypothesis
y = 1.00x + 0.00MSE: 0.00
Optimal Best Fit
y = 1.00x + 0.00MSE: 0.00
Understanding Linear Regression
Linear Regression is a fundamental algorithm in machine learning used to model the relationship between two variables. The goal is simple: find the straight line that best fits a set of data points. This line, often called the "regression line" or "hypothesis," is represented by the classic equation y = mx + c, where 'm' is the slope and 'c' is the y-intercept.
What Does "Best Fit" Mean? The Loss Function
How do we know which line is the "best"? We measure its error. For each data point, the vertical distance between the point and our line is called the residual or error. These are the red lines in the visualization. A good line should have small residuals.
To measure the total error for the entire dataset, we use a loss function. A common and effective one is the Mean Squared Error (MSE). Here's how it works:
For each data point, calculate the residual (error).
Square each residual. We do this so that negative and positive errors don't cancel each other out, and it also penalizes larger errors more heavily.
Calculate the average of all these squared residuals.
The "best" line is the one with the lowest possible MSE. Your goal is to find the values of 'm' and 'c' that minimize this loss.
Ordinary Least Squares (OLS)
While you can manually drag the line to try and find the best fit, there's a direct mathematical method to find the perfect 'm' and 'c'. This method is called Ordinary Least Squares (OLS). It's a formula that takes all the data points and instantly calculates the slope and intercept that guarantee the minimum possible Mean Squared Error. When you click the "Snap to Best Fit" button, you are seeing the OLS solution.
What the line is actually claiming
A fitted line is a claim in a sentence: "for every extra unit of x, y goes up by m — and when x is zero, y is c."
Fit house price against floor area and you might get price = 2.8 × area + 45. Read it back: every extra square metre is worth £2,800, and the model's starting point for a hypothetical zero-area house is £45,000. The first half is a genuinely useful business fact. The second half is a mathematical convenience — there are no zero-area houses, and the intercept is just where the line happens to cross the axis when extended beyond the data.
That distinction matters constantly. A slope is usually interpretable; an intercept usually is not, unless zero is a real, observed value of x.
With several features the equation grows terms but not ideas:
price = 2.8×area + 12.5×bedrooms − 0.9×age + 30
Each coefficient now means "the effect of this feature holding the others constant". Adding a bedroom is worth £12,500 for a house of the same floor area — which is a different and more careful statement than "houses with more bedrooms cost more".
Why squares, and not just distances
"Best fit" needs a definition before it can be computed. The obvious one — add up how far each point is from the line — turns out to be the wrong choice for three reasons.
Errors above and below the line cancel out if you add them raw, so a wildly wrong line through the middle scores zero. Absolute values fix that but have a corner at zero, which makes the calculus awkward and the solution not unique. Squares fix both: always positive, smooth everywhere, and differentiable.
Squaring also does something with real consequences: it punishes big misses disproportionately. Two points off by 5 contribute 50; one point off by 10 contributes 100. The fitted line will contort itself to reduce a single large error, which is why one outlier can visibly tilt a regression line and why you should always look at a scatter plot before trusting the coefficients.
There is a deeper justification too. If the errors around the true line are normally distributed, then the least-squares line is exactly the maximum-likelihood estimate — the line that makes the observed data most probable. That is why this particular definition of "best" has stuck for two centuries.
OLS has a formula, and gradient descent does not need it
Ordinary least squares is unusual among fitting procedures: the answer can be written down directly rather than searched for.
For a single feature, the slope and intercept come out as:
m = Σ(x − x̄)(y − ȳ) / Σ(x − x̄)² c = ȳ − m x̄
Two facts fall out of the second equation. The line always passes through the point (x̄, ȳ) — the mean of both variables. And once you know the slope, the intercept costs nothing to compute.
In matrix form for any number of features it becomes the normal equation, β = (XᵀX)⁻¹Xᵀy — one calculation, no iteration, no learning rate, and the answer is exact.
So why does anyone use gradient descent? Because inverting XᵀX costs roughly the cube of the number of features, and it fails outright when features are perfectly correlated, which makes the matrix non-invertible. Below a few thousand features, use the closed form. Above that, or when the data does not fit in memory, iterate.
from sklearn.linear_model import LinearRegression
model = LinearRegression().fit(X, y)
print(model.coef_, model.intercept_) # the slopes and the c
print(model.score(X, y)) # R-squared on the training data
The assumptions, and how to tell they are broken
Least squares always returns a line. Whether that line means anything rests on four assumptions, each with a visible symptom when it fails.
Assumption
What it means
How it fails visibly
Linearity
The real relationship is a straight line
Residuals form a curve, not a cloud
Independence
Errors are unrelated to each other
Residuals follow a pattern over time
Equal variance
Spread of errors is constant across x
Residual plot fans out like a funnel
Normal errors
Errors are roughly bell-shaped
Heavy tails on a Q-Q plot
The single most valuable diagnostic is the residual plot: predicted values on the x-axis, errors on the y-axis. A shapeless band around zero means the assumptions are holding well enough. A curve means you need a transformed or additional feature. A funnel means the error grows with the prediction, and modelling log(y) instead of y usually straightens it out.
Two more problems worth naming. Multicollinearity — features that are nearly duplicates — leaves the overall predictions fine but makes individual coefficients wild and unstable, flipping sign when you add a row. And extrapolation: a line fitted on houses of 50–200 square metres will confidently quote a price for a 900-square-metre house, and that number is fiction.
Solve it three ways and get the same line
The normal equation, scikit-learn, and gradient descent all land on the same coefficients. Seeing that makes the closed form stop being magic.
example_01.pyscikit-learn
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score, mean_squared_error
rng = np.random.default_rng(0)
n = 300
X = np.column_stack([rng.normal(0, 1, n), rng.normal(0, 2, n)])
true_w, true_b = np.array([2.5, -1.2]), 4.0
y = X @ true_w + true_b + rng.normal(0, 1.5, n)
Xb = np.column_stack([np.ones(n), X]) # bias column
# 1. the normal equation: w = (X'X)^-1 X'y
w_normal = np.linalg.solve(Xb.T @ Xb, Xb.T @ y)
print("normal equation b=%7.4f w=%s" % (w_normal[0], np.round(w_normal[1:], 4)))
# 2. scikit-learn
lr = LinearRegression().fit(X, y)
print("scikit-learn b=%7.4f w=%s" % (lr.intercept_, np.round(lr.coef_, 4)))
# 3. gradient descent, from scratch
w = np.zeros(3)
for step in range(4000):
grad = 2 * Xb.T @ (Xb @ w - y) / n
w -= 0.05 * grad
print("gradient descent b=%7.4f w=%s" % (w[0], np.round(w[1:], 4)))
print()
print("true values b=%7.4f w=%s" % (true_b, true_w))
print()
print("OLS minimises the sum of SQUARED residuals. nothing else has a lower one:")
best = ((y - Xb @ w_normal) ** 2).sum()
print(" the fitted line : %.4f" % best)
for nudge in (0.05, -0.05, 0.20):
alt = w_normal.copy(); alt[1] += nudge
print(" move w[0] by %+.2f : %.4f" % (nudge, ((y - Xb @ alt) ** 2).sum()))
print()
resid = y - lr.predict(X)
print("the residuals are what is left over:")
print(" they sum to almost exactly zero : %.10f" % resid.sum())
print(" and are uncorrelated with each x: %s"
% np.round([np.corrcoef(X[:, i], resid)[0, 1] for i in range(2)], 10))
print(" that is not luck -- it is what setting the derivative to zero means.")
print()
print("R^2 %.4f, RMSE %.4f (the noise we added had sd 1.50)"
% (r2_score(y, lr.predict(X)), np.sqrt(mean_squared_error(y, lr.predict(X)))))
print()
print("squaring is why one bad outlier moves the whole line:")
y2 = y.copy(); y2[0] += 60
print(" refit with one point moved by +60: w = %s"
% np.round(LinearRegression().fit(X, y2).coef_, 4))
print(" the other 299 points did not change.")
Output
Experiments to try
Use the interactive visualization to build your intuition.
Manual Fitting: Click "Generate New Data". Now, without using the "Snap" button, try to find the best fit yourself. Drag the purple (slope) and green (intercept) handles on the line. Watch the "Current Loss (MSE)" value in the left panel. Try to get it as low as possible. Notice how even small changes can dramatically affect the MSE.
See the Optimal Solution: Once you think you have a good fit, click the "Snap to Best Fit" button. The line will jump to the OLS solution, and you can see the "Optimal Loss" in the panel. How close were you?
The Impact of Outliers: Generate a new dataset. Now, click and drag one of the blue data points far away from the others. This point is now an "outlier". Click "Snap to Best Fit" again. Notice how the best-fit line is pulled towards the outlier. This demonstrates that OLS is sensitive to outliers because the squaring process in MSE gives large errors (like the one from the outlier) a very heavy weight.
Visualize the Loss: Check the "Show Optimal Line" box. Now your manual line (hypothesis) and the best-fit line are both visible. The "Live Loss" panel on the left shows you how far your current MSE is from the best possible MSE. This gives you a clear target: make your line match the amber-colored optimal line to achieve zero "Distance to Optimal".
Questions people ask
Do I need to scale the features? Not for the closed-form solution, which is unaffected by units. You do need it for gradient descent to converge sensibly, and for comparing coefficient sizes against each other.
Can linear regression fit curves? Yes, if you give it curved features. Add x² as a column and it fits a parabola — the model is linear in its parameters, not necessarily in the inputs. That is the whole idea behind polynomial regression.
What does a negative R² on the test set mean? That the model does worse than predicting the average. Usually overfitting, a data leak, or a distribution shift between train and test.
Is a p-value on a coefficient enough to claim an effect is real? No. It says the coefficient is unlikely to be zero given the model's assumptions. It says nothing about causation, and it is unreliable when the assumptions above are violated.
What if two features are almost identical? Drop one, combine them, or move to ridge regression, which handles correlated features gracefully by shrinking their coefficients together.
Is least squares robust to outliers? The opposite. Squaring makes it unusually sensitive. Huber regression or RANSAC are the standard robust alternatives when outliers are genuine and unavoidable.
Recap in one screen
The model is a straight line: a slope per feature, plus one intercept.
"Best" means the smallest sum of squared vertical distances — smooth, always positive, and heavily influenced by outliers.
OLS has an exact closed-form solution; gradient descent is for when that becomes too expensive.
The line always passes through the mean of x and the mean of y.
Read the residual plot before you read the coefficients, and never extrapolate past the data you fitted.
Check yourself
0 of 3
Answer without scrolling back up.
Ordinary least squares minimises the sum of:
Squaring makes every error positive so they cannot cancel, and it gives a smooth function with a closed-form solution. It also punishes large errors disproportionately.
Why does OLS react so strongly to a single far-off point?
An error of 10 contributes 100; an error of 1 contributes 1. One outlier can outweigh a hundred well-fitted points, which is why absolute-error methods are preferred when outliers are expected.
R-squared of 0.0 means:
R-squared compares your model against the trivial always-predict-the-mean baseline. Zero means you have matched that baseline and no more. It can even go negative if you do worse.
Cheat sheet
Linear Regression with OLS
Fit a line to a dataset by minimizing the Mean Squared Error (MSE). Drag the regression line manually to see how the residuals (red lines) change, or compute the exact best fit.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.