Students Professional
JA EN

#initialization

1 articles

01 ·Deep Learning Basics·★ MEMBER·PAPER·15 min read Weight Initialization and Regularization — What Lets Training Start, and What Keeps It Going The same architecture will train or refuse to train depending on nothing more than how large the random numbers in its weights were. Starting from how variance propagates, this piece derives Xavier and He initialization, then turns to weight decay and dropout as the conditions that keep training going — with interactive figures, PyTorch code and the mistakes that actually cost people days.