Take-home B — attention is two contractions

Attention answers question 5 from the video-pipeline discussion: which parts of a sequence matter most?

scores  = np.einsum('bid,bjd->bij', Q, K) / np.sqrt(dim)   # (4, 12, 12)
weights = softmax(scores, axis=-1)
output  = np.einsum('bij,bjd->bid', weights, V)            # (4, 12, 16)

mask = np.zeros((seq_len, seq_len)); mask[:, -3:] = -np.inf
weights_masked = softmax(scores + mask, axis=-1)   # padded positions get weight 0

scores is Chapter 2’s dot product; output is Chapter 2’s linear combination. Attention is two contractions built from ideas you have already read.

And the mask solves the variable-length problem from Part II: it is how real models handle sequences and videos of different lengths.