White-Balance Emulator for Color Augmentation

What Else Can Fool Deep Learning? Addressing Color Constancy Errors on Deep Neural Network Performance

Mahmoud Afifi¹ and Michael S. Brown^1,2
1York University    ²Samsung AI Center (SAIC) - Toronto

Project page - Paper - Supplementary Materials

Our augmentation method can accurately emulate realistic color constancy degradation. Existing color augmentation methods often generate unrealistic colors which rarely happen in reality (e.g., green skin or purple grass). More importantly, the visual appearance of existing color augmentation techniques does not well represent the color casts produced by incorrect WB applied onboard cameras, as shown below.

Quick start

1. Python:

1. Requirements: numpy & opencv-python

• pip install numpy
• pip install opencv-python

2. Run wbAug.py; examples:

• Process a singe image (generate ten new images and a copy of the given image):
  • python wbAug.py --input_image_filename ../images/image1.jpg
• Process all images in a directory (for each image, generate ten images and copies of original images):
  • python wbAug.py --input_image_dir ../images
• Process all images in a directory (for each image, generate five images without original images):
  • python wbAug.py --input_image_dir ../images --out_dir ../results --out_number 5 --write_original 0
• Augment all training images and generate corresponding ground truth files (generate three images and copies of original images):
  • python wbAug.py --input_image_dir ../example/training_set --ground_truth_dir ../example/ground_truth --ground_truth_ext .png --out_dir ../new_training_set --out_ground_truth ../new_ground_truth --out_number 3 --write_original 1

3. demo.py shows an example of how to use the WBEmulator module

2. Matlab:

1. Run install_.m
2. Try our demos:
   • demo_single_image to process signle image
   • demo_batch to process an image directory
   • demo_WB_color_augmentation to process an image directory and repeating the corresponding ground truth files for our generated images
   • demo_GUI (located in GUI directory) for a GUI interface
3. To use the WB augmenter inside your code, please follow the following steps:
   • Either run install_() or addpath to code/model directories:

    addpath('src');
    addpath('models'); 
    %or use install_()
   

• Load our model:

  load('synthWBmodel.mat'); %load WB_emulator CPU model --  use load('synthWBmodel_GPU.mat');  to load WB_emulator GPU model
  

• Run the WB emulator:

  out = WB_emulator.generate_wb_srgb(I, NumOfImgs); %I: input image tensor & NumOfImgs (optional): numbre of images to generate [<=10]
  

• Use the generated images:

  new_img = out(:,:,:,i); %access the ith generated image
  

Dataset

We used images from Set1 of the Rendered WB dataset to build our method.

In our paper, we introduced a new testing set that contains CIFAR-10 classes to evaluate trained models with different settings. This testing set contains 15,098 rendered images that reflect real in-camera WB settings. Our testing set is divided into ten directories, each includes testing images for one of CIFAR-10 classes. You can download our testing set from the following links: 32x32 pixels | 224x224 pixels | 227x227 pixels

MIT License

Publication

If you use this code or our dataset, please cite our paper:

Mahmoud Afifi and Michael S. Brown. What Else Can Fool Deep Learning? Addressing Color Constancy Errors on Deep Neural Network Performance. International Conference on Computer Vision (ICCV), 2019.

@InProceedings{Afifi_2019_ICCV,
author = {Afifi, Mahmoud and Brown, Michael S.},
title = {What Else Can Fool Deep Learning? Addressing Color Constancy Errors on Deep Neural Network Performance},
booktitle = {The IEEE International Conference on Computer Vision (ICCV)},
month = {October},
year = {2019}
}

Related Research Projects

• sRGB Image White Balancing:
  • When Color Constancy Goes Wrong: The first work for white-balancing camera-rendered sRGB images (CVPR 2019).
  • Color Temperature Tuning: A camera pipeline that allows accurate post-capture white-balance editing (CIC best paper award, 2019).
  • Interactive White Balancing: Interactive sRGB image white balancing using polynomial correction mapping (CIC 2020).
  • Deep White-Balance Editing: A multi-task deep learning model for post-capture white-balance editing (CVPR 2020).
• Raw Image White Balancing:
  • APAP Bias Correction: A locally adaptive bias correction technique for illuminant estimation (JOSA A 2019).
  • SIIE: A sensor-independent deep learning framework for illumination estimation (BMVC 2019).
  • C5: A self-calibration method for cross-camera illuminant estimation (arXiv 2020).
• Image Enhancement:
  • CIE XYZ Net: Image linearization for low-level computer vision tasks; e.g., denoising, deblurring, and image enhancement (arXiv 2020).
  • Exposure Correction: A coarse-to-fine deep learning model with adversarial training to correct badly-exposed photographs (CVPR 2021).
• Image Manipulation:
  • MPB: Image blending using a two-stage Poisson blending (CVM 2016).
  • Image Recoloring: A fully automated image recoloring with no target/reference images (Eurographics 2019).
  • Image Relighting: Relighting using a uniformly-lit white-balanced version of input images (Runner-Up Award overall tracks of AIM 2020 challenge for image relighting, ECCV Workshops 2020).
  • HistoGAN: Controlling colors of GAN-generated images based on features derived directly from color histograms (CVPR 2021).