Background

Inverse problems such as noise reduction, super-resolution, and inpainting can be formulated as the optimization task ${\displaystyle x^{*}=min_{x}E(x;x_{0})+R(x)}$, where ${\displaystyle x}$ is an image, ${\displaystyle x_{0}}$ a corrupted representation of that image, ${\displaystyle E(x;x_{0})}$ is a task-dependent data term, and R(x) is the regularizer. This forms an energy minimization problem.

Deep neural networks learn a generator/decoder ${\displaystyle x=f_{\theta }(z)}$ which maps a random code vector ${\displaystyle z}$ to an image ${\displaystyle x}$.

The image corruption method used to generate ${\displaystyle x_{0}}$ is selected for the specific application.

Specifics

In this approach, the ${\displaystyle R(x)}$ prior is replaced with the implicit prior captured by the neural network (where ${\displaystyle R(x)=0}$ for images that can be produced by a deep neural networks and ${\displaystyle R(x)=+\infty }$ otherwise). This yields the equation for the minimizer ${\displaystyle \theta ^{*}=argmin_{\theta }E(f_{\theta }(z);x_{0})}$ and the result of the optimization process ${\displaystyle x^{*}=f_{\theta ^{*}}(z)}$.

The minimizer ${\displaystyle \theta ^{*}}$ (typically a gradient descent) starts from a randomly initialized parameters and descends into a local best result to yield the ${\displaystyle x^{*}}$ restoration function.

Deep Neural Network Model

Typically, the deep neural network model for deep image prior uses a U-Net like model without the skip connections that connect the encoder blocks with the decoder blocks. The authors in their paper mention that "Our findings here (and in other similar comparisons) seem to suggest that having deeper architecture is beneficial, and that having skip-connections that work so well for recognition tasks (such as semantic segmentation) is highly detrimental."^[1]

Denoising

The principle of denoising is to recover an image ${\displaystyle x}$ from a noisy observation ${\displaystyle x_{0}}$, where ${\displaystyle x_{0}=x+\epsilon }$. The distribution ${\displaystyle \epsilon }$ is sometimes known (e.g.: profiling sensor and photon noise^[2]) and may optionally be incorporated into the model, though this process works well in blind denoising.

The quadratic energy function ${\displaystyle E(x,x_{0})=||x-x_{0}||^{2}}$ is used as the data term, plugging it into the equation for ${\displaystyle \theta ^{*}}$ yields the optimization problem ${\displaystyle min_{\theta }||f_{\theta }(z)-x_{0}||^{2}}$.

Super-resolution

Super-resolution is used to generate a higher resolution version of image x. The data term is set to ${\displaystyle E(x;x_{0})=||d(x)-x_{0}||^{2}}$ where d(·) is a downsampling operator such as Lanczos that decimates the image by a factor t.

Inpainting

Inpainting is used to reconstruct a missing area in an image ${\displaystyle x_{0}}$. These missing pixels are defined as the binary mask ${\displaystyle m\in \{0,1\}^{H\times V}}$. The data term is defined as ${\displaystyle E(x;x_{0})=||(x-x_{0})\odot m||^{2}}$ (where ${\displaystyle \odot }$ is the Hadamard product).

The intuition behind this is that the loss is computed only on the known pixels in the image, and the network is going to learn enough about the image to fill in unknown parts of the image even though the computed loss doesn't include those pixels. This strategy is used to remove image watermarks by treating the watermark as missing pixels in the image.

Flash–no-flash reconstruction

This approach may be extended to multiple images. A straightforward example mentioned by the author is the reconstruction of an image to obtain natural light and clarity from a flash–no-flash pair. Video reconstruction is possible but it requires optimizations to take into account the spatial differences.