gradient

Definition

The derivative of the loss with respect to each parameter: which direction, and how strongly, changing that parameter increases the error. Training takes a small step the opposite way, and backpropagation is what makes computing all of them at once affordable in a deep network. Keeping gradients from vanishing or exploding on the way back drives choices in both architecture — residual connections, normalization — and optimization, such as clipping.