multi-query

Definition

Multi-query attention keeps multiple query heads but collapses the key and value projections into a single shared pair. Because incremental decoding is bound by memory bandwidth rather than arithmetic, shrinking the KV cache this aggressively speeds generation substantially. The trade-off is a measurable quality drop on some tasks, which is why grouped-query attention — partial rather than total sharing — has become the more common choice.