Linear Regression is a fundamental algorithm in machine learning used to model the relationship between two variables. The goal is simple: find the straight line that best fits a set of data points. This line, often called the "regression line" or "hypothesis," is represented by the classic equation y = mx + c, where 'm' is the slope and 'c' is the y-intercept.
What Does "Best Fit" Mean? The Loss Function
How do we know which line is the "best"? We measure its error. For each data point, the vertical distance between the point and our line is called the residual or error. These are the red lines in the visualization. A good line should have small residuals.
To measure the total error for the entire dataset, we use a loss function. A common and effective one is the Mean Squared Error (MSE). Here's how it works:
- For each data point, calculate the residual (error).
- Square each residual. We do this so that negative and positive errors don't cancel each other out, and it also penalizes larger errors more heavily.
- Calculate the average of all these squared residuals.
The "best" line is the one with the lowest possible MSE. Your goal is to find the values of 'm' and 'c' that minimize this loss.
Ordinary Least Squares (OLS)
While you can manually drag the line to try and find the best fit, there's a direct mathematical method to find the perfect 'm' and 'c'. This method is called Ordinary Least Squares (OLS). It's a formula that takes all the data points and instantly calculates the slope and intercept that guarantee the minimum possible Mean Squared Error. When you click the "Snap to Best Fit" button, you are seeing the OLS solution.
Guided Experiments
Use the interactive visualization to build your intuition.
-
Manual Fitting:
Click "Generate New Data". Now, without using the "Snap" button, try to find the best fit yourself. Drag the purple (slope) and green (intercept) handles on the line. Watch the "Current Loss (MSE)" value in the left panel. Try to get it as low as possible. Notice how even small changes can dramatically affect the MSE.
-
See the Optimal Solution:
Once you think you have a good fit, click the "Snap to Best Fit" button. The line will jump to the OLS solution, and you can see the "Optimal Loss" in the panel. How close were you?
-
The Impact of Outliers:
Generate a new dataset. Now, click and drag one of the blue data points far away from the others. This point is now an "outlier". Click "Snap to Best Fit" again. Notice how the best-fit line is pulled towards the outlier. This demonstrates that OLS is sensitive to outliers because the squaring process in MSE gives large errors (like the one from the outlier) a very heavy weight.
-
Visualize the Loss:
Check the "Show Optimal Line" box. Now your manual line (hypothesis) and the best-fit line are both visible. The "Live Loss" panel on the left shows you how far your current MSE is from the best possible MSE. This gives you a clear target: make your line match the amber-colored optimal line to achieve zero "Distance to Optimal".