Padding and the validity mask

Real videos have different frame counts, but a batch tensor is rectangular. Take three real clips of length 4, 7 and 5 and pad them into one (3, 7, 135, 240, 3) order-5 batch.

Carry a Boolean (3, 7) validity mask alongside it, so valid.sum() == 16 — the 4 + 7 + 5 frames that are real. The padding then stays visible instead of being averaged into the data.

The same trick is how attention handles padded sequences: set the padded scores to -np.inf before the softmax.