kronecker-factored

Definition

Approximating a huge matrix as the Kronecker product of two much smaller ones. K-FAC — Kronecker-Factored Approximate Curvature — applies this to each layer's Fisher information matrix, factorising it into covariances of the layer's inputs and of its output gradients, which turns an intractable inverse into two small ones. It is the standard device for making second-order optimisation affordable in deep networks.