Mistake: keeping 99% of variance keeps the labels

“The shadow preserves almost everything, so it must be enough to classify.”

x = np.linspace(-30, 30, 100)
cloud = np.column_stack([np.repeat(x, 2), np.tile([-1., 1.], len(x))])
centered = cloud - cloud.mean(axis=0)
_, singular_values, axes = np.linalg.svd(centered, full_matrices=False)
shadow = centered @ axes[:1].T
assert singular_values[0]**2 / np.sum(singular_values**2) > .99
assert np.allclose(shadow[::2], shadow[1::2])

Opposite labels share the same wide horizontal coordinate. PCA keeps over 99% of the variance and gives each opposite-label pair the same score. The small vertical coordinate, the one PCA discards, separates every pair.

Retained variance is not retained information. Choose the representation by validation on the actual task within training data, then evaluate it on held-out data.