What Is a Diffusion Model? How Image-Generating AI Paints from Noise

Type in a few words and get a photorealistic image—tools like Stable Diffusion, Midjourney, and the DALL-E series have made image generation something anyone can do. Most of them are built on a shared piece of technology called the diffusion model.

The name sounds intimidating, but the core idea is surprisingly simple: deliberately add noise (a kind of static or “snow”) to a clean image, then learn to reverse that process and reconstruct the image from the noise. By learning this round trip, the model becomes able to produce pictures from nothing but random noise. This article explains what a diffusion model is, how it generates images, and how it differs from older approaches like GANs and VAEs, using academic papers and expert explanations and aimed at non-specialists. Reading it alongside our guide to large language models (LLMs), the foundation of generative AI, will give you a fuller map of the field.

What Is a Diffusion Model? Add Noise, Then Remove It

A diffusion model is one type of generative AI model used mainly in services that generate image data ². As LISKUL’s explainer puts it, it is “a kind of generative AI that can produce high-resolution images from scratch by deliberately adding noise to data such as images until it becomes random, then learning the process of gradually removing that noise” ³.

The starting point for this technique’s wide adoption in image generation was the 2020 paper “Denoising Diffusion Probabilistic Models” (DDPM). Written by Jonathan Ho, Ajay Jain, and Pieter Abbeel, the paper positioned diffusion models as “a class of latent variable models inspired by considerations from nonequilibrium thermodynamics” ¹ and showed that the approach yields “high quality image synthesis results” ¹. The prototype of diffusion models was proposed back in 2015, and the 2020 DDPM paper demonstrated its practical value in the field of image generation ².

How It Works: Two Processes, Forward and Reverse

At the heart of a diffusion model are two processes that run in opposite directions.

The first is the forward process (also called the diffusion process). AIsmiley’s explainer describes it as “the process of adding random noise to the original image data until it is finally converted into nothing but noise” ². You gradually add noise to a clean photo until it becomes meaningless static—in effect, a direction that destroys the image.

The second is the reverse process (the reverse diffusion process), which is “the process of removing noise from a Gaussian distribution to create an image” ². A trained model gradually moves from a noisy state toward real data ². What the AI actually learns is this “how to remove the noise.” Once it masters the steps for recovering a picture from static, it only needs to be given random noise at the start to generate a brand-new image from it.

In actual image generation, this reverse process follows a more concrete flow. According to LISKUL, it is a sequence of: (1) adding noise to training images over hundreds of steps until they become random static, (2) having a neural network predict and remove “how much noise should be removed at the current point,” (3) specifying the direction of generation with a prompt such as “a businessperson standing under a blue sky” (text conditioning), and (4) using acceleration algorithms like DDIM to shorten the time generation takes ³ (the actual generation time varies with hardware, resolution, and the number of steps). The DDPM paper itself notes that the model “naturally admit[s] a progressive lossy decompression scheme that can be interpreted as a generalization of autoregressive decoding” ¹, suggesting that it builds up an image by adding information little by little.

How It Differs from GANs and VAEs: Why Diffusion Became Mainstream

Diffusion is not the only approach to generating images with AI. Two notable predecessors are GANs and VAEs, and comparing them reveals why diffusion models came to be so widely used.

A GAN (generative adversarial network) trains by pitting a generator that creates “fakes” against a discriminator that judges “real or fake.” Performance can be high, but AIsmiley’s explainer points out that GANs suffer from “training instability and a lack of diversity” ²—meaning training can fail to converge, or the model can end up generating only similar-looking images.

A VAE (variational autoencoder) works by first compressing (encoding) data and then reconstructing it, but diffusion models have “no trainable parameters on the encoding side” ². This structural difference also affects the nature of the images produced.

In LISKUL’s comparison, diffusion models rank “high (resistant to mode collapse)” on training stability, while GANs are “unstable” and VAEs are “stable” ³. “Mode collapse” refers to a phenomenon where generated results become biased toward a limited set of patterns. Diffusion models are also said to offer high controllability, allowing adjustments mid-generation ³. The combination of stable training and the ability to generate diverse, high-quality images is likely the reason diffusion models became the mainstream of image-generating AI. Indeed, leading services such as Stable Diffusion (open source, free, and easy for anyone to use) and DALL-E 2 adopt diffusion models ².

What It’s Used For

Diffusion models go well beyond generating art and illustrations. LISKUL lists business use cases including marketing—mass-producing banners and landing-page assets in a short time—e-commerce product images that automatically generate color and background variations, 3D texture generation for games and the metaverse, faster design prototyping in manufacturing, and advertising A/B testing that efficiently produces hundreds to thousands of creative variations ³. Being able to create images from text compresses production workflows that used to cost significant time and money, which is where the business value lies.

The diffusion model is the “behind-the-scenes mechanism” of many image-generation tools we’ve covered, including Stable Diffusion, FLUX, and Nano Banana Pro. Some tools, like Midjourney, do not officially disclose their internal architecture, but they are widely believed to be based on diffusion models. Even when you’re tracking the new features and usability of individual tools, knowing that “painting from noise” sits at the root of them all makes it easier to understand why they can generate high-quality images so consistently.