Awesome Image Harmonization 
A curated list of resources including papers, datasets, and relevant links pertaining to image harmonization.
Contributing
Contributions are welcome. If you wish to contribute, feel free to send a pull request. If you have suggestions for new sections to be included, please raise an issue and discuss before sending a pull request.
Table of Contents
Online Demo
Try this online demo for image harmonization and have fun!
Leaderboard
The leaderboard of SOTA image harmonization methods can be found here.
Color Transfer
We summarize different color transfer strategies which could be used for image harmonization task here.
Papers
Supervised deep learning methods
- Wenyan Cong, Xinhao Tao, Li Niu, Jing Liang, Xuesong Gao, Qihao Sun, Liqing Zhang: "High-Resolution Image Harmonization via Collaborative Dual Transformations." CVPR (2022) [arXiv] [dataset] (high-resolution)
- Zhongyun Bao, Chengjiang Long, Gang Fu, Daquan Liu, Yuanzhen Li, Jiaming Wu, Chunxia Xiao: "Deep Image-based Illumination Harmonization." CVPR (2022) [arXiv] (rendered images)
- Yucheng Hang, Bin Xia, Wenming Yang, Qingmin Liao: "SCS-Co: Self-Consistent Style Contrastive Learning for Image Harmonization." CVPR (2022) [arXiv] [code]
- Ziyue Zhu, Zhao Zhang, Zheng Lin, Ruiqi Wu, Chunle Guo: "Image Harmonization by Matching Regional References." arXiv preprint arXiv:2204.04715 (2022) [arXiv]
- Jingtang Liang, Xiaodong Cun, and Chi-Man Pun: "Spatial-Separated Curve Rendering Network for Efficient and High-Resolution Image Harmonization." arXiv preprint arXiv:2109.05750 (2021) [arXiv] [code] (high-resolution)
- Wenyan Cong, Junyan Cao, Li Niu, Jianfu Zhang, Xuesong Gao, Zhiwei Tang, Liqing Zhang: "Deep Image Harmonization by Bridging the Reality Gap." arXiv preprint arXiv:2109.06671 (2021) [arXiv] [dataset] (rendered images)
- Zhongyun Hu, Ntumba Elie Nsampi, Xue Wang, Qing Wang: "NeSF: Neural Shading Field for Image Harmonization." arXiv preprint arXiv:2112.01314 (2021) [arXiv] (rendered images)
- Zonghui Guo, Dongsheng Guo, Haiyong Zheng, Zhaorui Gu, Bing Zheng, Junyu Dong: "Image Harmonization with Transformer." ICCV (2021) [pdf] [supp] [code]
- Yifan Jiang, He Zhang, Jianming Zhang, Yilin Wang, Zhe Lin, Kalyan Sunkavalli, Simon Chen, Sohrab Amirghodsi, Sarah Kong, Zhangyang Wang: "SSH: A Self-Supervised Framework for Image Harmonization." ICCV (2021) [pdf] [supp] [arXiv] [code]
- Jun Ling, Han Xue, Li Song, Rong Xie, Xiao Gu: "Region-Aware Adaptive Instance Normalization for Image Harmonization." CVPR (2021) [pdf] [supp] [arXiv] [code]
- Zonghui Guo, Haiyong Zheng, Yufeng Jiang, Zhaorui Gu, Bing Zheng: "Intrinsic Image Harmonization." CVPR (2021) [pdf] [supp] [code]
- Wenyan Cong, Li Niu, Jianfu Zhang, Jing Liang, Liqing Zhang: "BargainNet: Background-Guided Domain Translation for Image Harmonization." ICME (2021) [arXiv] [code]
- Konstantin Sofiiuk, Polina Popenova, Anton Konushin: "Foreground-aware Semantic Representations for Image Harmonization." WACV (2021) [pdf] [supp] [arXiv] [code]
- Guoqing Hao, Satoshi Iizuka, Kazuhiro Fukui: "Image Harmonization with Attention-based Deep Feature Modulation." BMVC (2020) [pdf] [supp] [code]
- Wenyan Cong, Jianfu Zhang, Li Niu, Liu Liu, Zhixin Ling, Weiyuan Li, Liqing Zhang: "DoveNet: Deep Image Harmonization via Domain Verification." CVPR (2020) [pdf] [supp] [arXiv] [code].
- Xiaodong Cun, Chi-Man Pun: "Improving the Harmony of the Composite Image by Spatial-Separated Attention Module." IEEE Trans. Image Process. 29: 4759-4771 (2020) [pdf] [arXiv] [code]
- Yi-Hsuan Tsai, Xiaohui Shen, Zhe Lin, Kalyan Sunkavalli, Xin Lu, Ming-Hsuan Yang: "Deep Image Harmonization." CVPR (2017) [pdf] [supp] [arXiv] [code]
Unsupervised deep learning methods
- Anand Bhattad, David A. Forsyth: "Cut-and-Paste Neural Rendering." arXiv preprint arXiv: 2010.05907 (2020) [arXiv] [supp]
- Fangneng Zhan, Shijian Lu, Changgong Zhang, Feiying Ma, Xuansong Xie:"Adversarial Image Composition with Auxiliary Illumination." ACCV (2020) [pdf] [arXiv]
- Bor-Chun Chen, Andrew Kae: "Toward Realistic Image Compositing With Adversarial Learning." CVPR (2019) [pdf]
- Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, Alexei A. Efros: "Learning a Discriminative Model for the Perception of Realism in Composite Images." ICCV (2015) [pdf] [arXiv] [code]
Traditional methods
- Shuangbing Song, Fan Zhong, Xueying Qin, Changhe Tu: "Illumination Harmonization with Gray Mean Scale." Advances in Computer Graphics. CGI (2020) [pdf]
- Su Xue, Aseem Agarwala, Julie Dorsey, Holly E. Rushmeier: "Understanding and improving the realism of image composites." ACM Trans. Graph. 31(4): 84:1-84:10 (2012) [pdf]
- Kalyan Sunkavalli, Micah K. Johnson, Wojciech Matusik, Hanspeter Pfister: "Multi-scale image harmonization." ACM Trans. Graph. 29, 4 (2010) [pdf]
- Jean-François Lalonde, Alexei A. Efros: "Using Color Compatibility for Assessing Image Realism." ICCV (2007) [pdf] [code]
Datasets
- iHarmony4: It contains four subdatasets: HCOCO, HAdobe5k, HFlickr, Hday2night, with a total of 73,146 pairs of unharmonized images and harmonized images. [pdf] [link]
- GMSDataset: It contains 183 images with image resolution of 1940*1440. It consists of 16 different objects and for each object, one source image and 11 target images in different background scenes and illumination conditions are captured. [pdf] [link] (access code: ekn2)
- HVIDIT: A dataset built upon VIDIT (Virtual Image Dataset for Illumination Transfer) dataset for image harmonization. It contains 3007 images of 276 scenes for training and 329 images of 24 scenes for testing. [pdf] [link]
- RHHarmony: A rendered human harmonization dataset, which contains 15,000 ground-truth rendered images and has the potential to generate 135,000 composite rendered images. [pdf] [link]
- RealHM: A Real-world HarMonization dataset, which contains 216 real composite images with manually harmonized outputs. [pdf] [link]
- ccHarmony: An image harmonization dataset constructed based on the images captured with color checker (cc). It contains 10 synthetic composite images for each of 426 foregrounds from 350 real images, leading to 4260 pairs of synthetic composite images and ground-truth real images. [pdf] [link]
Related Topics
Painterly harmonization
- Hwai-Jin Peng, Chia-Ming Wang, Yu-Chiang Frank Wang: "Element-Embedded Style Transfer Networks for Style Harmonization." BMVC (2019) [paper]
- Fujun Luan, Sylvain Paris, Eli Shechtman, Kavita Bala: "Deep Painterly Harmonization." Computer graphics forum (2018) [arXiv] [code]
Inharmonious region localization
- Jing Liang, Li Niu, Penghao Wu, Fengjun Guo, Teng Long: "Inharmonious Region Localization by Magnifying Domain Discrepancy." AAAI (2022)
- Jing Liang, Li Niu, Liqing Zhang: "Inharmonious Region Localization." ICME (2021) [arXiv] [code]
Video harmonization
- Xinyuan Lu, Shengyuan Huang, Li Niu, Wenyan Cong, Liqing Zhang: "Deep Video Harmonization with Color Mapping Consistency." IJCAI (2022) [arXiv] [dataset&code]
- Haozhi Huang, Senzhe Xu, Junxiong Cai, Wei Liu, Shimin Hu: "Temporally Coherent Video Harmonization Using Adversarial Networks." IEEE Trans. Image Process. 29: 214-224 (2020) [pdf] [arXiv]