Why it matters
Generating a coherent image requires satisfying global structure and local detail at once. Diffusion handles this by starting from noise and refining, so early steps commit to composition and later steps to texture. The engineering problem is control: adherence to the prompt, reproducibility across seeds, and the cost of the many denoising steps that make an output good enough to ship.
Mental model
Image generation is sculpting from a block of marble: broad strokes remove most of the material, then fine tools shape the surface. The metaphor breaks because the model revisits the whole image at every step, so a late adjustment can still alter the overall composition.
How it works
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. A synthetic latent grid resolving from noise to a schematic landscape over an illustrative 20-step schedule. The prompt, guidance 7.5 and 20 steps are authored; the grid is a 24 × 16 class map, not a diffusion latent, and no model generated the image.
What to notice. The schedule is coarse-to-fine: by 30% the layout is committed, and the remaining steps refine texture inside it. A late change of composition is therefore unlikely, while a late change of surface detail is expected. Guidance pushes adherence and repetition together; the decode step has its own artefacts on fine text and faces.
Text version
A 24 by 16 schematic landscape with sky, sun, hills, a red-roofed house, a tree and a path. Six steps show the prompt encoding with a still-noisy latent, step 0 pure noise, step 6 at 30 percent with the composition settled, step 14 at 70 percent with edges and texture, the decoded image, and a guidance note. The denoise view mixes a coarse version of the class map with deterministic noise: early frames are blocky and noisy, later frames crisp.
- Encode the prompt with the text encoder the model was trained against, and keep prompt templates consistent with its training captions.
- Choose the number of denoising steps and the sampler together, since step counts interact with sampler behaviour.
- Set guidance scale for the adherence and diversity trade-off, and test both extremes on your prompt set.
- Fix the seed and record it, so a reported result can be reproduced exactly.
- Use negative prompts and structural conditioning such as depth or edges when composition matters more than variety.
- Evaluate with a fixed prompt suite and human review, since automatic scores correlate weakly with perceived quality.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| High guidance | Strong prompt adherence | Saturated, repetitive outputs | Prompts are unusual or need variety |
| Many steps | Detail and stability | Latency per image | Interactive use |
| Latent diffusion | Fast at high resolution | Autoencoder artefacts on fine text | The task needs pixel-exact output |
| Pixel-space diffusion | No autoencoder loss | Compute and memory | Resolution is high |
Good for
Strong prompt adherence
Cost
Saturated, repetitive outputs
Fails when
Prompts are unusual or need variety
- Prompts are unusual or need variety
- Interactive use
- The task needs pixel-exact output
- Resolution is high
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: High guidance is good for Strong prompt adherence, costs Saturated, repetitive outputs, and fails when Prompts are unusual or need variety. Many steps is good for Detail and stability, costs Latency per image, and fails when Interactive use. Latent diffusion is good for Fast at high resolution, costs Autoencoder artefacts on fine text, and fails when The task needs pixel-exact output. Pixel-space diffusion is good for No autoencoder loss, costs Compute and memory, and fails when Resolution is high
Failure modes
- Prompt ignored. The image follows a frequent concept instead of the stated one. Debug: run the prompt with several seeds and check whether any seed obeys it, which separates prompt failure from sampling variance.
- Autoencoder artefacts. Small text and faces are mangled even with good prompts. Debug: decode a known image through the autoencoder and compare it with the original.
- Guidance overdrive. Colours clip and compositions repeat across seeds. Debug: sweep guidance at a fixed seed and compare output diversity.
In production
Record prompt, seed, sampler, step count and model revision with every generated asset, so a complaint can be reproduced. Cap generation concurrency and queue length, since diffusion saturates accelerators.
Common misconceptions
People think: The model retrieves images it has seen.
Actually: It denoises from noise; memorisation is a data problem, not the mechanism.
People think: A higher guidance scale means a better image.
Actually: Past a point it narrows diversity and produces oversaturated artefacts.
Interview angle
Denoising in a latent space is the mechanism, and the autoencoder is part of the quality budget. A strong answer treats guidance as a dial with a cost at both ends and insists on recording the seed. Weak answers describe text-to-image as a single forward pass and treat the prompt as the only variable.
Sources
- Ho et al., Denoising Diffusion Probabilistic Models — the denoising objective behind diffusion models.
- Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models — denoising in a compressed latent space.
- Ho & Salimans, Classifier-Free Diffusion Guidance — the guidance trade-off between adherence and diversity.