Mistake: training teaches DistMult which way a relation goes

“DistMult scores (France, contains, Paris) and (Paris, contains, France) the same at first, but with enough facts training will break the tie.”

rng_claim = np.random.default_rng(19)
e_paris, e_france, w_contains = rng_claim.standard_normal((3, 8))
france_contains_paris = np.sum(e_france * w_contains * e_paris)
paris_contains_france = np.sum(e_paris * w_contains * e_france)
assert np.isclose(france_contains_paris, paris_contains_france)

The score is a sum of products and multiplication commutes: the two scores are equal for every embedding, at any dimension, after any amount of training. No data can teach a model a distinction its score function cannot express.

In the notebook’s toy world — 3 countries, 9 cities, 27 districts — DistMult separates none of the 36 contains facts from their reversals and gives both a probability near one half. TuckER, with a dense core, gets every one right.