Mistake: a tiny residual proves reliable coefficients

“Both fits are almost exact, so their coefficients must agree.”

X = np.array([[1., 1.], [1., 1.000001]])
a, b = np.array([1., 1.]), np.array([2., 0.])
y = X @ a
assert np.linalg.norm(X @ a - y) == 0
assert np.isclose(np.linalg.norm(X @ b - y), 1e-6)
assert np.isclose(np.linalg.norm(a - b), np.sqrt(2))

The coefficients differ by about 1.414, while the predictions differ by only 0.000001. Nearly dependent columns make this possible.

Inspect conditioning and coefficient sensitivity as well as fit. QR avoids forming X.T @ X; it does not remove the problem’s sensitivity.