Adding matrices, and broadcasting

Two matrices of the same shape add element by element: C = A + B, with Cᵢ,ⱼ = Aᵢ,ⱼ + Bᵢ,ⱼ. A scalar can be added to every element, or multiply every element: D = a · B + c.

Deep learning also allows a less conventional shorthand: adding a vector to a matrix. In C = A + b, with Cᵢ,ⱼ = Aᵢ,ⱼ + bⱼ, the vector b is added to each row of A, with no need to build a matrix holding a copy of b in every row first. This implicit copying is called broadcasting, the same rule NumPy and PyTorch follow.