Back

Least Squares as Projection

Fitting a line to scattered points is a projection in disguise. Stack the measured heights into a vector $\mathbf{b}$ and the model values $mx_i + c$ into the column space of $A = [\,\mathbf{x}\ \ \mathbf{1}\,]$: no line passes exactly through noisy data, so $\mathbf{b}$ lies off that plane, and the best fit is the point in the plane closest to it, the orthogonal projection. Closest means the residual $\mathbf{r} = \mathbf{b} - A\hat{\mathbf{x}}$ is perpendicular to the plane, $A^\top\mathbf{r} = 0$, which is exactly the normal equations $A^\top A\,\hat{\mathbf{x}} = A^\top\mathbf{b}$. For a line they solve to $\hat m = S_{xy}/S_{xx}$ through the centroid of the data. The red bars are the residuals; drag the blue handle to tilt the line off the best fit and watch every bar, and the total, grow. The panel below is the projection picture itself: the data vector b, its foot p in the column-space plane, and the residual e joining them, standing perpendicular to the plane only at the least-squares fit and leaning over as you tilt. Drag the gold points to reshape the data and the fit re-solves at once.

Figure 1. Least squares as projection. Top: the data (gold), the best-fit line (blue) through the centroid (green cross) with the vertical residuals (red); tilting the line with its handle shows the optimal line as a green ghost. Bottom: the same fit as a projection in the column space, the data vector b, its foot p = A x-hat in the plane, and the residual e = b - p, perpendicular to the plane exactly at the least-squares fit (A^T e = 0, the normal equations) and leaning, growing, when the line is tilted off it. Method: the normal equations, slope = Sxy / Sxx. Source: Strang, Introduction to Linear Algebra, 5th ed., Sec. 4.3.

WHAT TO TRY

  • Drag the blue handle to tilt the line off the best fit: the red residual bars and the total SSR grow, and in the panel below the residual e tips away from perpendicular and lengthens.
  • Snap back to the best fit: e stands perpendicular to the column-space plane (the right-angle mark), which is $A^\top e = 0$, the normal equations.
  • Drag a gold point far from the others (an outlier) and watch the squared residual pull the whole line toward it.
  • Notice the line always passes through the green centroid, whatever the slope.