What heads cost

Heads enable distinct attention patterns — each head can attend differently in its own feature space.

They are not a speed-up. At fixed D, heads do not reduce the leading O(B · S² · D) dot-product work: H heads of D_k = D / H features do the same multiplications as one head of D.

And memory grows with them: the scores occupy B · H · S² elements, quadratic in the sequence length.