lora

Definition

Low-Rank Adaptation: a fine-tuning method that freezes the pre-trained weights and learns only a low-rank factorization of the weight update. Because the trainable and stored parameters shrink by orders of magnitude, large models can be adapted on consumer hardware and the result shipped as a small adapter file rather than a full checkpoint. It has become the default for both LLM fine-tuning and community-shared image-generation styles.

Explainers using this term

• A Practical Map of Image Generation — SD, ControlNet, and Applying LoRALoRA: Low-Rank Adaptation of Large Language Models
• Singular Value Decomposition and Low-Rank Approximation — the Math Behind LoRALoRA: Low-Rank Adaptation of Large Language Models
• Paper Walkthrough: Macaron-V1 — A Frozen Base plus a Mixture of LoRAs, Built to Keep Learning After LaunchMacaron-V1: Towards Open Continual Learning with Self-Improvement and Mixture-of-LoRA