softmax

Definition

Turns a vector of real scores into a positive distribution summing to one, by exponentiating each entry and dividing by their sum. It produces class probabilities at the output of a classifier and the weights inside attention. The name suggests a smooth maximum, but what it actually smooths is argmax — it returns a distribution over positions, not a value — and a temperature divisor controls how sharply it concentrates on the top score.