multi-head

Definition

Running several attention operations in parallel, each in its own lower-dimensional subspace, then concatenating and projecting the results. The motivation is that a single attention distribution can only emphasize one kind of relation at a time, whereas separate heads can specialize. The per-head dimension is the model width divided by the head count, so adding heads redistributes capacity rather than adding computation.