An Introduction to Visual Generation

About this keynote: This post summarizes a survey-style introduction to visual generation, tracing how deep generative models extend from unconditional image synthesis toward increasingly rich conditional controls and advanced applications.

---

Preliminary

Deep Generative Models

In early days, people from different groups explored visual generative models via denoising diffusion probabilistic models (DDPM) [Ho et al., 2020] and score-based generative models (NCSN) [Song and Ermon, 2019]. Later on, they were interpreted under the same framework called Score-SDE [Song et al., 2021]. After that, novel frameworks such as Flow Matching [Lipman et al., 2023] emerged, which are out of the scope of Score-SDE.

Nowadays, a promising unified framework called stochastic interpolants [Albergo et al., 2025] unifies most generative modeling frameworks. However, there is still a lack of common sense to name all these model groups with a unified category. Hence, for simplicity and clarity, we refer to them as deep generative models throughout this presentation.

Outline

1. Unconditional/Class-Conditional Image Generation
2. Text-to-Image Generation
3. Controllable Image Generation
4. Image Editing
5. Image-to-Image Translation
6. Video Generation
7. Advanced Applications
   • Inversion Task
   • Representation Extractor
   • Generalist Vision Learner
   • 3D/4D Generation & World Modeling
   • etc.

One Training Objective

“There is only one precise way of presenting the laws, and that is by means of differential equations. They have the advantage of being fundamental and, so far as we know, precise.”

— Richard P. Feynman

Many deep generative models share one training recipe [Lai et al., 2025]: fit a network to a noisy version of the data, weighted over time.

\[\mathcal{L}(\phi) := \mathbb{E}_{\mathbf{x}_0,\,\boldsymbol{\epsilon}} \left[ \underbrace{ \mathbb{E}_{p_{\mathrm{time}}(t)} }_{\text{time distribution}} \left[ \underbrace{ \omega(t) }_{\text{time weighting}} \underbrace{ \left\| \mathrm{NN}_{\phi}\!\bigl(\mathbf{x}_t, t\bigr) - \bigl(A_t \mathbf{x}_0 + B_t \boldsymbol{\epsilon}\bigr) \right\|_2^2 }_{\text{MSE part}} \right] \right]\]

The design space can be summarized along four axes:

• (A) Noise schedule in the forward process of (\mathbf{x}_t) via (\alpha_t) and (\sigma_t).
• (B) Prediction types of (\mathrm{NN}_{\phi}) and regression targets (\bigl(A_t \mathbf{x}_0 + B_t \boldsymbol{\epsilon}\bigr)).
• (C) Time-weighting function (\omega(\cdot) : [0, T] \to \mathbb{R}_{\geq 0}).
• (D) Time distribution (p_{\mathrm{time}}).

---

Unconditional Image Generation

This section covers (p(\mathbf{x})) and class-conditional (p(\mathbf{x} \mid c)), where (c) is a discrete class label (e.g., an ImageNet category)—not a text prompt or visual control signal.

---

Text-to-Image Generation

Synthesize images from natural language descriptions. Latent Diffusion Models (LDM; Stable Diffusion) [Rombach et al., 2022] led the era of latent modeling in research and AIGC in business.

---

Controllable Image Generation

Steer generation with spatial, structural, or semantic controls. ControlNet [Zhang et al., 2023] injects auxiliary conditions (e.g., edges, pose) into a frozen diffusion U-Net.

---

Image Editing

Move from text-to-image (p(x\mid c_T)) to instruction-guided editing (p(x\mid c_T, c_I)), where (c_I) is the input image. InstructPix2Pix [Brooks et al., 2023] concatenates the input image latent with the noisy latent along the channel dimension, introducing (<)0.1M trainable parameters.

---

Image-to-Image Translation

Translate images across domains by learning a diffusion bridge between source and target distributions. DDBM [Zhou et al., 2024] generalizes score-based diffusion to Schrödinger-bridge image translation.

---

Video Generation

Extend image generation to coherent temporal synthesis with efficient diffusion transformers. SANA-Video [Chen et al., 2026] uses a deep compression VAE for spatiotemporal latent modeling, together with block linear DiT and autoregressive block training for efficient text-to-video generation.

---

Advanced Applications

Beyond standard synthesis: inversion, representation learning, and world modeling.

Inversion Task

Solve inverse problems with a pre-trained diffusion generative model—no further training required. DDRM [Kawar et al., 2022] restores images from degraded observations via diffusion priors.

Representation Extractor

Use pre-trained diffusion models as representation extractors. DiffSeg [Tian et al., 2024] performs zero-shot segmentation from Stable Diffusion attention maps.

Generalist Vision Learner

Build unified vision systems from large-scale generative pre-training. Vision Banana [Gabeur et al., 2026] shows image generators can be generalist vision learners for both generation and understanding.

3D/4D Generation & World Modeling

Generate navigable spatiotemporal worlds and learn interactive environment models from video. YUME 1.5 [Mao et al., 2025] extends YUME [Mao et al., 2025] with text-controlled exploration, trained on Sekai [Li et al., 2025].

---

Takeaways

1. We start from an unconditional image generator (p(x)) and progressively add more conditions to it. Technically, this is representation alignment: new condition signals—text, segmentation maps, source images, etc.—are injected into a pre-trained denoiser.
2. Besides their powerful generative capability, deep generative models are also representation learners [Yang and Wang, 2023], often matching or even surpassing SSL methods such as MoCo on downstream recognition.
3. Joint generative modeling (p(x,y)) can natively unify generation and understanding via Bayes’ rule: marginalize to obtain (p(x)=\int p(x,y)\,dy) for synthesis, and derive (p(y\mid x)=p(x,y)/p(x)) for discriminative inference—without training a separate head. A more robust foundation model built on this principle remains under-explored.
4. We are still on the way toward a generalist, task-agnostic, any-to-any foundation model [Gabeur et al., 2026], [Zuo et al., 2025].
5. Building strong generative models is promising and already feasible, but open challenges remain in efficient training, sampling, alignment, strong generalization, and more.
6. Today’s visual generation can exceed human perceptual limits—how do we properly evaluate it?

---

References

---