Study decks

Vectors, matrices, norms, eigenvalues, the SVD, the pseudoinverse and PCA: the linear algebra the tensors workshop builds on, from Deep Learning, Ch. 2.

The four words you need before the workshop starts, the nine real datasets it runs on, and the shapes you will meet all day.

The vocabulary the whole workshop runs on — tensor, axis, order, shape, slice, fiber, unfolding, contraction — and the two meanings of “rank”.

Read every axis as a question: what does this axis count? Batch against time, the five axis letters, and padding with a validity mask.

Select by name, not by number. Fancy indexing, Boolean masks, broadcasting — and the zero-variance pixels that make NaN a finding, not a bug.

Changing a tensor’s shape is not the same as moving its axes. HWC, CHW, NHWC, NCHW — and the reshape that runs without error and scrambles the image.

Every pipeline decision chooses what information is kept, transformed or discarded. Shapes at five stages, sampling loss, and where a camera axis goes.

One rule: an index that disappears after the arrow is summed over. Grayscale, trace, matrix products and a 1797x1797 similarity matrix, all in one syntax.

Does an exact inverse exist? If not, what useful answer does the pseudoinverse give instead? Solved on 20,433 real California housing districts.

Use the current state to create the next state, then repeat. Fibonacci, power iteration and a real airline forecast — the structure of a recurrent network.

Which factorization do I reach for, and what does it cost? LU, QR, Cholesky, eigendecomposition, SVD and NMF, each asked the same two questions.

PCA generalized to every axis. One factor matrix per mode plus a small core — and a real taxi tensor whose hour factor peaks at the evening rush.

CP, Tucker, Tensor Train and t-SVD: four bargains about which structure to keep. Choose the method from the structure, then the rank from the loss.

The idea connecting the workshop, five take-homes (PCA, attention, CP, Cholesky, audio denoising) and the four deep dives opened live.

Correlation, convolution, transposed convolution, deconvolution: one kernel, four operations, and a real blurred photograph partly recovered.

A CP component is one vector per axis. Fit it by alternating least squares, keep it non-negative, and watch neural trials sort by a label CP never saw.

Squared error is an assumption. Swap in a Poisson or Bernoulli loss, fit real Chicago crime counts, and catch a component that is an artifact of the file.

Covariance, correlation, eigenvectors and SVD on real satellite patches; does PCA help a classifier? Then multilinear PCA and a spiked tensor signal.

Split Q, K and V into heads without losing a token: reshape then transpose, scores over keys, √D_k, merging back, and why shape tests lie.

Compress a feature map by truncated SVD of an unfolding: rank-k reconstruction, energy as squared amplitude, factor storage, and what 95% misses.

CP or Tucker? One object separates them, the core. A knowledge graph, a ResNet-18 kernel and a photo show why CP compresses and Tucker predicts.

Matrix multiplication is a tensor of 0s and 1s, and every exact CP split into R terms is an algorithm with R multiplications. Strassen, TensorGame, AlphaTensor.