Project DelphiTensors Workshop

CP is four convolutions, Tucker-2 is three

A 2-D convolution layer’s weights form a 4-way tensor: out-channels × in-channels × height × width. Substitute a rank-R CP and move each sum inward, and the layer becomes four layers (Lebedev et al. 2015):

  1. a 1×1 conv, 256 → R channels
  2. a 3×1 depthwise conv on each channel
  3. a 1×3 depthwise conv on each channel
  4. a 1×1 conv, R → 256 channels

Depthwise (groups=R) means each channel is convolved on its own: a rank-one term never mixes with another — the superdiagonal core again. The chain stores 518R weights; R = 36 stores 32× fewer numbers than the dense kernel.

Tucker-2 compresses only the two channel modes (Kim et al. 2016) and becomes three layers: a 1×1 conv, an ordinary 3×3 conv whose kernel is the core, and another 1×1 conv.