Neural Networks for Low Level Image Processing

Until recently, machine learning (ML) and neural network (NN) have been mainly used in high level vision tasks, such as image segmentation, classification and object detection. Low level image processing tasks such as denoising, demosaicing, white balance still mainly rely on signal processing based methods which need a lot of expert knowledge to desige the filters. There are usually a long list of filters in a complete image processing pipeline running on a dedicated image signal processor(ISP). Designing and optimizing a list of filters require a lot of effort. Apple has a patent with more than 500 pages covering only a part of the full ISP pipeline!

In the past few years, two new developments are changing the image processing field. One trend is that more and more neural network based methods have been proposed to handle a part or the whole image processing pipeline and achieved fascinating performance in term of image quality and processing speed. The other trend is that neural network chip becomes more and more popular at various mobile platform, such as the latest Apple A11 Bionic chip and Huawei Kirin 970 chip. I believe in the near future, neural network based methods will play important roles at low level image processing tasks, and neural network computing units will become an integrated part of the image signal processor.

This is a personal collection of research works using neural network for low level image processing. I will try to keep it up to date. You are welcome to contribute to this list. Papers of significance are marked in bold. My comments are marked in italic.

Table of Contents

• Review and comments
• Color constancy
• Denoising
• Demosaicing
• Superresolution
• Automatic adjustment
• Artefacts removal
• Pipeline
• Image quality evaluation
• Others

Review and comments

• Deep, Deep Trouble
  • This is an interesting comment made by Michael Elad about the impact of Deep Learning on image processing, mathematics and humanity.
• Deep learning for image/video processing
  • Dr. Huang Yu's summary of DL on image/video processing.

Color constancy

• Deep Specialized Network for Illuminant Estimation (ECCV, 2016, CUHK)
• Single and Multiple Illuminant Estimation Using Convolutional Neural Networks (TIP, 2017, Italy)
  • A three-stage method for illuminant estimation from RAW images
• Recurrent Color Constancy (ICCV, 2017, TUT)
  • Recurrent network for temporal color constancy using multiple frames instead of single frame
• FC4: Fully Convolutional Color Constancy with Confidence-weighted Pooling (CVPR, 2017, THU)
  • CNN based method using image patches
• Natural Image Illuminant Estimation via Deep Non-negative Matrix Factorisation (IET-IP, 2017, OceanUofChina)

Denoising

• Image denoising with multi-layer perceptrons (arXiv, 2012)
  • This is the very first paper using NN to image denoising tasks.
• Can a Single Image Denoising Neural Network Handle All Levels of Gaussian Noise? (SPL, 2014, France)
  • This paper proposal a way to apply NN on image denoising with different noise level of Gaussian noise.
• Deep convolutional architecture for natural image denoising (WCSP, 2015, ZJU)
• Learning Deep CNN Denoiser Prior for Image Restoration (arXiv, 2017, HKPolyU)
• Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising (TIP, 2017, HKPolyU)
• FFDNet: Toward a Fast and Flexible Solution for CNN based Image Denoising (arXiv, 2017, HKPolyU)
• Image Restoration: From Sparse and Low-Rank Priors to Deep Priors (IPM, 2017, HKPolyU)
  • This is a review paper.
• Dilated Deep Residual Network for Image Denoising (arXiv, 2017, SIU)
• IDEAL: Image DEnoising AcceLerator (ACM, 2017, UToronto&Algolux)
  • A NN based approximations of BM3D on an NN accelerator
• Deep Burst Denoising (arXiv, 2017, UCL&Facebook)

Demosaicing

• Demosaicing using artificial neural networks (SPIE, 2000)
• A multilayer neural network for image demosaicking (ICIP, 2012)
• Deep Joint Demosaicking and Denoising (Siggraph, 2016, MIT&Adobe)
  • This paper propose to use more difficult patches for the training.
  • Data and code are available on Github.
• Deep Image Demosaicking using a Cascade of Convolutional Residual Denoising Networks (arXiv, 2018, Russia)

Automatic adjustment

From the publications, we can find that Adobe has done a lot of work pushing the usage of machine learning in low-level image processing especially automatic photo adjustment.

• Learning Photographic Global Tonal Adjustment with a Database of Input / Output Image Pairs (CVPR, 2011, MIT&Adobe)
  • Adjustment personalization
• Automatic Photo Adjustment Using Deep Neural Networks (ACM, 2016, UIUC&Adobe&Microsoft)
  • This technique is more accurate and supports local edits.

Superresolution

Super-resolution is one of the areas that NN has been applied extensively and achieved great success.

• Deep Networks for Image Super-Resolution with Sparse Prior (ICCV, 2015, UIUC)
• Image superresolution using deep convolutional networks (TPAMI, 2015, HKUST)
• Accurate image super-resolution using very deep convolutional networks (CVPR, 2016, Seoul Univ.)
• Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network (CVPR, 2016, ICL)
  • An order of magnitude faster than previous CNN-based methods
• Image Super-Resolution via Deep Recursive Residual Network (CVPR, 2017, MSU)
• Photo-realistic single image super-resolution using a generative adversarial network (CVPR, 2017, Twitter)

Artefacts removal

• Deep Generative Adversarial Compression Artifact Removal (arXiv, 2017, UoFlorence)
• Real-time Deep Video Deinterlacing (arXiv, 2017, CUHK)

Pipeline

• Learning the image processing pipeline (arXiv, 2016, Stanford)
  • Propose to learn the filter parameters by ML
• Learning Adaptive Parameter Tuning for Image Processing (arXiv, 2016, UCLA&Nvidia)
• Deep Image Demosaicking using a Cascade of Convolutional Residual Denoising Networks (arXiv, 2018, Technion)
• DeepISP: Learning End-to-End Image Processing Pipeline(arXiv, 2018, Technion)

Image quality evaluation

• Deep Learning for Blind Image Quality Assessment (2017, Unimib)

Others

• Fast Image Processing with Fully-Convolutional Networks (ICCV, 2017, Intel)
  • Operator approximation to accelerate image processing tasks
• Learning Proximal Operators:Using Denoising Networks for Regularizing Inverse Imaging Problems (arXiv, 2017, TUM)
  • This paper proposes a way to use NN for the the general fidelity plus regularization optimization problem which may find applications in many problems.
• Learning to See in the Dark (CVPR,2018, UIUC&Intel)
  • This paper presented a very useful dark imaging dataset and an end-to-end training of a fully-convolutional network based on U-Net.