Denoising Diffusion Restoration Models

Bahjat Kawar¹, Michael Elad¹, Stefano Ermon², Jiaming Song²
¹Technion, ²Stanford University

ArXiv PDF Code

DDRM uses pre-trained DDPMs for solving general linear inverse problems. It does so efficiently and without problem-specific supervised training.

Abstract

Many interesting tasks in image restoration can be cast as linear inverse problems. A recent family of approaches for solving these problems uses stochastic algorithms that sample from the posterior distribution of natural images given the measurements. However, efficient solutions often require problem-specific supervised training to model the posterior, whereas unsupervised methods that are not problem-specific typically rely on inefficient iterative methods. This work addresses these issues by introducing Denoising Diffusion Restoration Models (DDRM), an efficient, unsupervised posterior sampling method. Motivated by variational inference, DDRM takes advantage of a pre-trained denoising diffusion generative model for solving any linear inverse problem. We demonstrate DDRM's versatility on several image datasets for super-resolution, deblurring, inpainting, and colorization under various amounts of measurement noise. DDRM outperforms the current leading unsupervised methods on the diverse ImageNet dataset in reconstruction quality, perceptual quality, and runtime, being 5× faster than the nearest competitor. DDRM also generalizes well for natural images out of the distribution of the observed ImageNet training set.

Demo Images

Show me

images that are

white Gaussian noise of σ=0.05

Original
Degraded
Restored

All images are restored using a fast 20-step unconditional DDRM

Variational Inference

We observe \(\mathbf{y} = \mathbf{Hx} + \mathbf{z}\), where \( \mathbf{x} \) is the original image, \( \mathbf{H} \) is the degradation operator, and \( \mathbf{z} \sim \mathcal{N}(0, \sigma_{\mathbf{y}}^2 \mathbf{I}) \) is additive white Gaussian noise with known variance. In order to restore the original image, we define DDRM as a Markov chain \( \mathbf{x}_T \to \mathbf{x}_{T-1} \to \ldots \to \mathbf{x}_{1} \to \mathbf{x}_{0} \) conditioned on \( \mathbf{y} \). \( \mathbf{x}_T \) will be approximated by a known distribution, and the diffusion process of the Markov chain will lead to \( \mathbf{x}_0 \), our final output for the estimated restored image. We define this Markov chain in the spectral space of the degradation operator \( \mathbf{H} \), using elements from its singular value decomposition (SVD). We then construct an evidence lower bound (ELBO) on the maximum likelihood objective for DDRM. We show its relation to the objective detailed in DDPM, therefore motivating using their pre-trained models in DDRM.

Quantitative Results

We evaluate DDRM on 1,000 images from the ImageNet dataset (256 × 256), and measure the average PSNR and KID (\(\times10^3\)). We compare DDRM (with 20 steps) with other unsupervised methods that work in reasonable time and can operate on ImageNet.

	PSNR ↑	KID ↓	NFEs ↓
DGP	22.70	27.60	1500
RED	26.16	21.21	500
SNIPS	34.32	0.49	1000
DDRM	35.64	0.71	20

DDRM can also be extended for nonlinear inverse problems such as JPEG artifact correction. See paper and code.

BibTeX

@inproceedings{kawar2022denoising,
    title={Denoising Diffusion Restoration Models},
    author={Bahjat Kawar and Michael Elad and Stefano Ermon and Jiaming Song},
    booktitle={Advances in Neural Information Processing Systems},
    year={2022}
}