parameter

Definition

A number inside the model whose value is set by training — the weights and biases that gradient descent updates. Their count is the field's default measure of model size, because it fixes capacity and, directly, how much memory the weights occupy. Distinguish hyperparameters such as learning rate or batch size: those are chosen by the practitioner around the training run and are never updated by gradients.

Explainers using this term

• Hyperparameter Search — Hunches, Grids, and Bayesian OptimizationRandom Search for Hyper-Parameter Optimization
• Mixture of Experts (MoE) from Scratch — Routing and Load Balancing in the Switch TransformerSwitch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity