Padding against fixed-frame sampling
Clips range from 30 seconds to 4 hours, and a batch tensor is rectangular. There are two strategies, and they trade off differently:
- Pad to the longest clip and carry a validity mask. Nothing is thrown away, but a 4-hour clip in the batch makes every 30-second clip mostly padding.
- Sample a fixed number of frames from every clip. The batch stays small and even, but the sampling discards most of the long clips.
An invented or wasted value in that tensor represents a moment that was never recorded, or a moment that was recorded and then dropped.