A Practical Map of Image Generation — SD, ControlNet, and Applying LoRA

A single map for anyone about to touch image generation for the first time: the four boxes of latent diffusion, where the prompt actually takes effect, locking down composition with ControlNet, fine-tuning with LoRA, and the licensing layers that quietly cause the most damage.

LoRA: Low-Rank Adaptation of Large Language Models

Image generation is not one model

The first thing that hits you is the vocabulary. SD1.5, SDXL, FLUX, ControlNet, LoRA, CFG, samplers, VAE. They all sound like concepts at the same level, and they are not — each one lives in a completely different part of the system.

A kitchen analogy helps. The base model is your ingredients and equipment. The prompt is the order. ControlNet is a mold that says "plate it exactly like this." LoRA is a small recipe card that teaches one specific flavor. The sampler and step count are the heat and the cooking time. All of them are in play at once, but each one acts somewhere different.

This article is a map of those "somewheres." The individual theory lives in other articles; here we only walk through which part sits where, and what you will actually be adjusting on the job — with no prior knowledge assumed.

The big picture: four boxes

Today's mainstream image models (the Stable Diffusion family, the FLUX family) are built from four boxes.

1. Text encoder — turns a string into a sequence of numbers (embeddings). CLIP and T5 live here
2. VAE (encoder/decoder) — compresses the image into a small "latent image," then expands it back at the end
3. Denoising network — estimates the noise sitting on the latent image and subtracts it. A UNet or a Transformer
4. Sampler — the procedure deciding how many times, and with what step sizes, box 3 gets called

Box 2 is where most of the leverage is. A 512×512 color image is $512 \times 512 \times 3 = 786{,}432$ numbers, but the VAE in Stable Diffusion 1.x/2.x/XL shrinks height and width by a factor of 8 and carries 4 channels instead of 3 — so $64 \times 64 \times 4 = 16{,}384$ numbers. Denoising runs in a space roughly 48× smaller.

This is "latent diffusion." Pixel space is full of fine detail that carries no meaning a human would notice; hand that to the VAE, and let the diffusion model handle only the coarse structure that decides what the picture is. That division of labor is the direct reason image generation started running on consumer GPUs.

The training objective is nearly identical to plain diffusion.

Symbol by symbol: $z$ is the compressed latent image, $c$ is the prompt turned into embeddings, $\epsilon$ is the noise we actually mixed in, $t$ is the timestep standing for how much noise, $z_t$ is $z$ with timestep-$t$ noise mixed in, and $\epsilon_\theta$ is the network being trained. Put in words: shrink the picture, stir noise into it, show the model the prompt, and make it guess which noise you stirred in. That's all. Anything you can guess you can subtract, and once you've subtracted it all, a picture is left. The add-then-subtract idea itself is covered in Diffusion Models from Scratch.

Where the prompt actually acts

People call prompts "spells," but the mechanism is concrete. The text becomes a sequence of embeddings, and every block of the denoising network meets it at a cross-attention layer.

$$\mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V, \quad Q = W_Q z,\; K = W_K \tau(c),\; V = W_V \tau(c)$$

The whole line, in words: hold each question the image asks ($Q$) up against the name tag each prompt token is waving ($K$), and carry back a blend of the tokens' contents ($V$) in proportion to how well they matched. The division by $\sqrt{d}$ is there because dot products grow with the embedding dimension $d$, and without it the softmax would collapse onto a single token.

Here's the point: the query $Q$ comes from the image side, while the key $K$ and value $V$ come from the text side. Each position in the latent image asks "what am I supposed to be painting here?", and each prompt token answers "I'm holding this information." Their agreement is measured by a dot product, and softmax turns those scores into proportions.

So word order and emphasis changing the picture isn't superstition. Change a token's embedding and you change the dot products; change the dot products and you change how much of each token gets listened to.

One more mechanism applies only at generation time: CFG (classifier-free guidance). Each step calls the network twice — once with the prompt shown, once without.

$\varnothing$ means "no prompt," and $s$ is the knob your tool labels CFG Scale. The equation says: take the difference between with-prompt and without-prompt — the direction in which the prompt is pulling — and push $s$ times further along it. Turn it up and the model follows instructions more literally; turn it up too far and colors burn out while variety dies. Negative prompts are the same machinery: put blurry, watermark into the "no prompt" slot instead of an empty string, and you get pushed away from that. The internals are taken apart in CFG and Samplers.

Choosing a model: what to judge on

The base model is the first decision on any job. Five axes cover it.

Native resolution. SD 1.5 was trained around 512px, SDXL around 1024px. Ask for a size far from what it was trained on and you get artifacts — figures duplicating limbs, that sort of thing. If you need bigger, generate first and run an upscaler after.

VRAM. If it doesn't fit, you're getting by on quantization or sequential offloading, and paying for it in speed.

Ecosystem. Easy to overlook, and it matters. ControlNets and LoRAs are weights tied to a specific base architecture — an SD1.5 LoRA will not load onto SDXL. How much of that material already exists translates directly into how fast you work.

Text rendering and instruction following. Rendering legible text inside the image, or holding the spatial relationships between several subjects, tends to be stronger in newer generations of models.

Licensing. Covered below, but this is where the most rework comes from — checking it last. It belongs at the front of your selection process.

Control: reach for the weakest lever first

When the picture isn't what you wanted, the available moves form a ladder of strength, and the rule is to start weak. The stronger the intervention, the harder it is to tell what broke when something breaks.

1. Prompt — add words, change the order, move things into the negative
2. Seed and sampler — the same settings with different initial noise give a different picture. Lock a good seed, then tune everything else
3. img2img — put a rough sketch into latent space and start denoising partway through. denoising strength decides how far in you start; near 0 stays faithful to the original
4. inpaint — repaint only a masked region. Hands and faces get fixed here
5. ControlNet — pin the composition itself with line art, depth, or pose
6. IP-Adapter family — feed a reference image's "feel" in as a condition

ControlNet is the one that imports structure. The mechanism is direct: clone the encoder half of the denoising network, freeze the original weights, and train only the clone. At the clone's entrance and exit sit "zero convolutions" — layers initialized to all zeros.

$$y_c = \mathcal{F}(x; \Theta) + \mathcal{Z}\big( \mathcal{F}(x + \mathcal{Z}(c; \Theta_{z1}); \Theta_c); \Theta_{z2} \big)$$

$\mathcal{F}$ is the original block, $\Theta$ the frozen original weights, $\Theta_c$ the clone's weights, $\mathcal{Z}$ a zero convolution, and $c$ the condition image (line art, a depth map). Read aloud: add a correction derived from the condition on top of the original output — except the correction's entrance and exit start at zero, so at step one of training the model behaves exactly as it did before. Because it starts by doing nothing, you can't wreck the huge pretrained model on the first gradient step, and the conditioning can be learned from relatively little data.

Three things you actually touch: the preprocessor (Canny / Depth / OpenPose / Scribble and friends), the strength (controlnet_conditioning_scale), and the active range (from what percent to what percent of generation it applies).

Fine-tuning: applying a LoRA

Retraining a whole model is out of reach on both data and GPU hours. LoRA (Low-Rank Adaptation) is what you use instead. It started as a method for large language models, but image generation is where it became an everyday tool. The idea is one line: instead of updating the weight matrix $W_0$ itself, add the product of two skinny matrices as a delta.

$W_0$ is the frozen original weight, $A$ squeezes the input down to $r$ dimensions, $B$ expands it back to the original width, $r$ is the rank, and $\alpha$ is the scale factor called alpha. $B$ is initialized to zero, so the delta is exactly 0 when training starts — the same idea as ControlNet's zero convolutions.

Read it in words: leave the original weight alone, and run the input down one extra detour beside it — narrowed to $r$ dimensions, then widened back to full size — scaling that detour by $\alpha/r$ before adding it in. Since the detour is the only thing being trained, $r$ itself caps how much change the adapter can express. Dividing by $r$ is what keeps the strength of the effect from shifting every time you change the rank.

Count the parameters and the lightness explains itself. A $1280 \times 1280$ weight is about 1.64 million numbers; a rank-8 LoRA is two $1280 \times 8$ matrices, about 20 thousand — under 1/80th. That's why the base model is gigabytes while a LoRA file is megabytes to a few hundred megabytes.

As the figure shows, a higher rank buys expressiveness at the cost of overfitting risk and file size. The right value differs depending on whether you're teaching only a style or the consistency of one specific subject. (For the original paper's experiments and evidence, see LoRA, Explained.)

Loading one takes very little code.

from diffusers import StableDiffusionXLPipeline
import torch

pipe = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16
).to("cuda")

pipe.load_lora_weights("./my-style.safetensors")   # add only the delta
image = pipe(
    "a ceramic teapot on a wooden table",
    negative_prompt="blurry, watermark",
    guidance_scale=6.0,                      # the s in equation (2)
    num_inference_steps=30,
    cross_attention_kwargs={"scale": 0.7},   # how much of equation (3)'s delta to mix in
).images[0]

Lowering scale (written <lora:name:0.7> in the web UIs) softens the effect. When stacking several LoRAs, settle each one's strength individually so the total doesn't run away from you.

On the training side the knobs are rank, alpha, learning rate, step count — and how you write the captions, which matters most. Collect what you want learned into a trigger word, and explicitly caption what you don't want learned (background, clothing). Whatever goes uncaptioned is what gets absorbed into the trigger.

Licensing: check it first

More projects are damaged here than by anything technical, and you need to keep three layers separate.

1. The license on the model weights. It differs completely per model. Stable Diffusion 1.x/2.x ship under CreativeML Open RAIL-M and SDXL 1.0 under OpenRAIL++-M — both carry use-based restrictions, which makes them a different animal from a normal open-source license. With FLUX.1, [schnell] is Apache-2.0 while [dev] is a non-commercial license: terms can split within a single family. "It's on Hugging Face" does not mean "free for commercial use."

2. The license on add-on weights — LoRAs, ControlNets. Distributors often attach their own terms (no commercial use, no redistribution). A commercially usable base does not rescue output made with a non-commercial LoRA.

3. Training data and subject rights. A LoRA trained on a real person, an existing character, or one artist's style can raise likeness and copyright questions well before licensing enters the picture. A license file saying "OK" only speaks to the rights the model's distributor holds.

On top of that, rights in the generated output are treated differently across jurisdictions and are still moving. Route commercial work through legal review; this article is not legal advice. The practical defense is simple: record the base model, the add-on weights, the license URLs, and the date you obtained them, alongside the output. Not being able to reconstruct what made a given image is the dangerous state.

How this shows up on the job

Who touches it, and when. A concept artist on a game or app spinning out variations before committing to production. An e-commerce team bringing product-photo background swaps and banner assets in-house. An agency putting several rough directions in front of a client quickly. An ML platform engineer standing up an internal generation API and watching its quality and cost.

Parameters you will actually type. guidance_scale (CFG Scale), num_inference_steps, denoising strength, seed, controlnet_conditioning_scale, a LoRA's scale plus its rank / alpha, and sampler names (Euler a, DPM++ 2M Karras, and so on). Locally the tools are ComfyUI or the Automatic1111-style web UIs; from code, Hugging Face diffusers is the de facto standard.

Pitfalls that turn into incidents.

• Not recording seed and versions. "We can't reproduce that one image" is the single most common failure. Set up the habit on day one: model name and hash, LoRAs and their strengths, seed, sampler, steps, CFG — kept together.
• Straying too far from native resolution. Asking a 512px-trained model for 1024px directly breaks composition. Generate, then upscale.
• Assuming more CFG is better. Past a certain point saturation clips and detail collapses. When output ignores your instructions, the cause is usually an ambiguous prompt, not insufficient CFG.
• Stacking too many LoRAs. Three or four at once and the deltas interfere; isolating the cause becomes impossible.
• Mismatched VAE. Point a model at a VAE it wasn't paired with and everything goes gray or muddy. It's the classic answer to "why do the colors look wrong?"
• Leaving the license check for last. Discovering a non-commercial license after delivery means rebuilding the LoRA and everything downstream of it.

How it comes up in a design review. "What are ControlNet's zero convolutions for?" (So the output is unchanged at the start of training, and the pretrained model isn't destroyed.) "What happens when you raise a LoRA's rank?" (More expressiveness and a bigger file, with more overfitting risk.) "The output still ignores the prompt after raising CFG — what do you check next?" (Prompt ambiguity, token-length truncation, an over-stuffed negative prompt, the base model's instruction-following ability.)

Wrapping up

• Image generation is four boxes — text encoder, VAE, denoising network, sampler — and each component acts on a different box
• The prompt acts through cross-attention; CFG extrapolates along the with-prompt minus without-prompt difference
• ControlNet adds structure, LoRA adds style or subject. Both are designed to start at zero so the pretrained model survives
• Reach for the weakest lever first; starting from the strong end destroys your ability to diagnose
• Read licensing in three layers — weights, add-on weights, subject rights — and read it first, not last

With the map in place, the knobs are next. What governs generation's "strength" and "speed" continues in CFG and Samplers; the root of the add-then-subtract idea is in Diffusion Models from Scratch.