mixture-of-lora

Definition

Applying the mixture-of-experts idea to low-rank adapters: keep several LoRAs trained for different tasks or domains, and let a router decide which ones to apply, and with what weights, per input or per token. Compared with a full MoE whose experts are complete feed-forward blocks, the added parameters are tiny, so a single base model can carry many specialisations and they remain cheap to train and ship separately.

Explainers using this term

• Paper Walkthrough: Macaron-V1 — A Frozen Base plus a Mixture of LoRAs, Built to Keep Learning After LaunchMacaron-V1: Towards Open Continual Learning with Self-Improvement and Mixture-of-LoRA