Natural/color image segmentation

A paper list of unsupervised natural/color image segmentation.

*Last updated: 2022/05/05

Update log

2020/March - update all of recent papers and make some diagram about history of natural/color image segmentation.
2020/July - update some recent papers and codes.
2020/August - update some recent papers and codes.
2020/December - update some recent papers and codes.
2021/February - update some recent papers and codes.
2022/January - update some recent papers and codes.
2022/May - update some recent papers and codes.

Table of Contents

• Datasets
• Metrics
• Performance tables
• Paper list

Datasets

The papers related to datasets used mainly in natural/color image segmentation are as follows.

• [BSDS300] Berkeley segmentation dataset 300 includes 300 natural images and the ground truth data. Each image has a fixed size of 481×321 pixels.

• [BSDS500] Berkeley segmentation dataset 500 is an improved version of BSDS300 dataset. the BSDS500 contains 500 natural images. Each image is annotated by 5 different people on average.

• [MSRC] Microsoft Research Cambridge v2 dataset contains 591 images and 23 object classes with accurate pixel-wise labeled images.

Metrics

The papers related to metrics used mainly in natural/color image segmentation are as follows.

• [PRI] Probabilistic rand index
• [VOI] Variation of information
• [GCE] Global consistency error
• [BDE] Boundary displacement error
• [SC] Segmentation Cover
• [F] F-measure

Performance tables

Speed is related to the hardware spec(e.g. CPU, GPU, RAM, etc), so it is hard to make an equal comparison. We select six indexs namely PRI, VoI, GCE, BDE, SC, and F-measure to make comparison. The closer the segmentation result is to the ground truth, the higher the PRI, SC and F-measure are, and the smaller the other three measures are.

BSDS300 (*test set)

BSDS500 (*test set)

MSRC (*test set)

Paper list

• [Ncut] Jianbo, S. and J. Malik (2000). "Normalized cuts and image segmentation." IEEE Transactions on Pattern Analysis and Machine Intelligence 22(8): 888-905. [code]
• [JSEG] Deng, Y. and B. S. Manjunath (2001). "Unsupervised segmentation of color-texture regions in images and video." IEEE Transactions on Pattern Analysis and Machine Intelligence 23(8): 800-810.
• [Meanshift] Comaniciu, D. and P. Meer (2002). "Mean shift: a robust approach toward feature space analysis." IEEE Transactions on Pattern Analysis and Machine Intelligence 24(5): 603-619.
• [FH] Felzenszwalb, P. F. and D. P. Huttenlocher (2004). "Efficient graph-based image segmentation." International Journal of Computer Vision 59(2): 167-181. [code]
• [MNCut] Cour, T., et al. (2005). Spectral segmentation with multiscale graph decomposition. IEEE Conference on Computer Vision and Pattern Recognition 1124-1131.
• [Canny-UCM] Arbelaez, P. (2006). Boundary extraction in natural images using ultra-metric contour maps. IEEE Conference on Computer Vision and Pattern Recognition Workshop: 182-182.
• [gPb-Hoiem] Hoiem, D., et al. (2007). "Recovering surface layout from an Image." International Journal of Computer Vision 75: 151–172.
• [NTP] Jingdong, W., et al. (2008). Normalized tree partitioning for image segmentation. IEEE Conference on Computer Vision and Pattern Recognition: 1-8.
• [Quickshift] Vedaldi, A. and S. Soatto (2008). Quick shift and kernel methods for mode seeking. European Conference on Computer Vision: 705-718. [code]
• [CTM] Yang, A. Y., et al. (2008). "Unsupervised segmentation of natural images via lossy data compression." Computer Vision and Image Understanding 110(2): 212-225. [code]
• [SDTV] Donoser, M., et al. (2009). Saliency driven total variation segmentation. IEEE International Conference on Computer Vision: 817-824.
• [PRIF] Mignotte, M. (2010). "A label field fusion Bayesian model and its penalized maximum rand estimator for image segmentation." IEEE Transactions Image Processing 19(6): 1610-1624. [code]
• [SuperParsing] Tighe, J. and S. Lazebnik (2010). SuperParsing: Scalable nonparametric image parsing with superpixels. European Conference on Computer Vision: 352-365.
• [Context-sensitive] Xiang, B., et al. (2010). "Learning context-sensitive shape similarity by graph transduction." IEEE Transactions on Pattern Analysis and Machine Intelligence 32(5): 861-874.
• [gPb-owt-ucm] Arbelaez, P., et al. (2011). "Contour detection and hierarchical image segmentation." IEEE Transactions on Pattern Analysis and Machine Intelligence 33(5): 898-916. [code]
• [LRR & MLAP] Cheng, B., et al. (2011). Multi-task low-rank affinity pursuit for image segmentation. IEEE International Conference on Computer Vision: 2439-2446.
• [TBES] Mobahi, H., et al. (2011). "Segmentation of natural images by texture and boundary compression." International Journal of Computer Vision 95(1): 86-98.
• [TPG] Yang, X. and L. J. Latecki (2011). Affinity learning on a tensor product graph with applications to shape and image retrieval. IEEE Conference on Computer Vision and Pattern Recognition: 2369-2376.
• [Co-transduction] Bai, X., et al. (2012). "Co-transduction for shape retrieval." IEEE Transactions Image Processing 21(5): 2747-2757.
• [Joint Kernel] Kim, J., et al. (2012). Joint kernel learning for supervised image segmentation. Asian Conference on Computer Vision.
• [SAS] Zhenguo, L., et al. (2012). Segmentation using superpixels: A bipartite graph partitioning approach. IEEE Conference on Computer Vision and Pattern Recognition: 789-796. [code]
• [FusionTP] Zhou, Y., et al. (2012). Fusion with diffusion for robust visual tracking. Advances in Neural Information Processing Systems.
• [Corr-Cluster] Kim, S., et al. (2013). "Task-specific image partitioning." IEEE Transactions Image Processing 22(2): 488-500.
• [LFPA] Kim, T. H., et al. (2013). "Learning full pairwise affinities for spectral segmentation." IEEE Transactions on Pattern Analysis and Machine Intelligence 35(7): 1690-1703.
• [HAFCM] Tan, K. S., et al. (2013). "Novel initialization scheme for Fuzzy C-Means algorithm on color image segmentation." Applied Soft Computing 13(4): 1832-1852.
• [RFHA] Tan, K. S., et al. (2013). "Color image segmentation using adaptive unsupervised clustering approach." Applied Soft Computing 13(4): 2017-2036.
• [L0-GRAPH] Wang, X., et al. (2013). A Graph-cut approach to image segmentation using an affinity graph based on ℓ_0-sIncorporating texture information into region-based unsupervised image segmentation using textural superpixelsparse representation of features. IEEE International Conference on Image Processing: 4019-4023.
• [WCP] Wang, X., et al. (2013). Graph-based image segmentation using weighted color patch. IEEE International Conference on Image Processing: 4064-4068.
• [HPCQ] Cho, S. I., et al. (2014). "Human Perception-based image segmentation using optimising of colour quantisation." IET Image Processing 8(12): 761-770.
• [CCM] Gu, X., et al. (2014). Improving superpixel-based image segmentation by incorporating color covariance matrix manifolds. IEEE International Conference on Image Processing: 4403-4406.
• [SASFinal] Hsu, C.-Y., et al. (2014). Incorporating texture information into region-based unsupervised image segmentation using textural superpixels. IEEE International Conference on Image Processing: 4323-4327.
• [ATP] Jingdong, W., et al. (2014). "Regularized tree partitioning and its application to unsupervised image segmentation." IEEE Transactions Image Processing 23(4): 1909-1922.
• [HO-CC] Kim, S., et al. (2014). "Image segmentation using higher-order correlation clustering." IEEE Transactions on Pattern Analysis and Machine Intelligence 36(9): 1761-1774.
• [FMBFM] Mignotte, M. and C. Hélou (2014). "A Precision-recall criterion based consensus model for fusing multiple segmentations." International Journal of Signal Processing, Image Processing and Pattern Recognition 7(3): 61-82. [code]
• [RIS-HL] Wu, J., et al. (2014). Reverse image segmentation: A high-level solution to a low-level task. British Machine Vision Conference.
• [LDCQP] Chakeri, A. and L. O. Hall (2015). Large data clustering using quadratic programming: A comprehensive quantitative analysis. IEEE International Conference on Data Mining Workshop: 806-813.
• [CCP] Fu, X., et al. (2015). Robust image segmentation using contour-guided color palettes. IEEE International Conference on Computer Vision: 1618-1625. [code]
• [FSHA] Verdoja, F. and M. Grangetto (2015). Fast superpixel-based hierarchical approach to image segmentation. International Conference on Image Analysis and Processing: 364-374.
• [GL-GRAPH] Wang, X., et al. (2015). "A Global/local affinity graph for image segmentation." IEEE Transactions Image Processing 24(4): 1399-1411. [code]
• [FBTS] Yuan, J., et al. (2015). "Factorization-based texture segmentation." IEEE Transactions Image Processing 24(11): 3488-3497.
• [GTD] Yu Z., et al. (2015). Generalized transitive distance with minimum spanning random forest. International Joint Conference on Artificial Intelligence (IJCAI): 2205-2211.
• [ISM] Li, X., et al. (2016). "An integrated similarity metric for graph-based color image segmentation." Multimedia Tools and Applications 75(6): 2969-2987.
• [LSI] Dong, L., et al. (2016). "LSI: Latent semantic inference for natural image segmentation." Pattern Recognition 59: 282-291
• [SHST] Gu, X., et al. (2016). A Hierarchical segmentation tree for superpixel-based image segmentation. International Conference on Image and Vision Computing New Zealand(IVCNZ): 1-6.
• [ISCC] Huang, D., et al. (2016). "Ensembling over-segmentations: From weak evidence to strong segmentation." Neurocomputing 207: 416-427.
• [GCEBFM] Khelifi, L. and M. Mignotte (2016). GCE-based model for the fusion of multiples color image segmentations. IEEE International Conference on Image Processing: 2574-2578.
• [MOBFM] Khelifi, L. and M. Mignotte (2016). A multi-objective approach based on TOPSIS to solve the image segmentation combination problem. IEEE International Conference on Pattern Recognition: 4220-4225.
• [2DKLMPSO] Zhao, X., et al. (2016). "A multilevel image thresholding segmentation algorithm based on two-dimensional K–L divergence and modified particle swarm optimization." Applied Soft Computing 48: 151-159.
• [LMS_GLA] Cho, H., et al. (2017). "Image segmentation using linked mean-shift vectors and global/local attributes." IEEE Transactions on Circuits and Systems for Video Technology 27(10): 2132-2140.
• [MCG] Pont-Tuset, J., et al. (2017). "Multiscale combinatorial grouping for image segmentation and object proposal generation." IEEE Transactions on Pattern Analysis and Machine Intelligence 39(1): 128-140.
• [UHIS_FEM] Yin, S., et al. (2017). "Unsupervised hierarchical image segmentation through fuzzy entropy maximization." Pattern Recognition 68: 245-259.
• [CDS] Fu, X.,et al. (2017). "Image segmentation using contour, surface, and depth cues." IEEE International Conference on Image Processing: 81-85.
• [AS-SR] Mahjoub, M. A., et al. (2017). "Adaptive strategy for superpixel-based region-growing image segmentation." Journal of Electronic Imaging 26(06):1-24.
• [HVAA] Lee, H. S. and Y. Hwan Kim (2018). Human visual attention analysis-based image segmentation using Color Histogram. International SoC Design Conference (ISOCC): 76-77.
• [FRFCM] Lei, T., et al. (2018). "Significantly fast and robust fuzzy C-means clustering algorithm based on morphological reconstruction and membership filtering." IEEE Transactions on Fuzzy Systems 26(5): 3027-3041. [code]
• [H_+R_Better] Li, K., et al. (2018). "Iterative image segmentation with feature driven heuristic four-color labeling." Pattern Recognition 76: 69-79.
• [2DNLMeKGSA] Mittal, H. and M. Saraswat (2018). "An optimum multi-level image thresholding segmentation using non-local means 2D histogram and exponential Kbest gravitational search algorithm." Engineering Applications of Artificial Intelligence 71: 226-235.
• [AMR] Lei, T., et al. (2019). "Adaptive morphological reconstruction for seeded image segmentation." IEEE Transactions Image Processing 28(11): 5510-5523. [code]
• [SFFCM] Lei, T., et al. (2019). "Superpixel-based fast fuzzy C-means clustering for color image segmentation." IEEE Transactions on Fuzzy Systems 27(9): 1753-1766. [code]
• [EA-GRAPH] Sun, G., et al. (2019). "An enhanced affinity graph for image segmentation." IEICE Transactions on Information and Systems E102.D(5): 1073-1080.
• [AASP-GRAPH] Zhang, Y., et al. (2019). "An adaptive affinity graph with subspace pursuit for natural image segmentation," IEEE International Conference on Multimedia and Expo: 802-807. [code]
• [DSFCM_N] Zhang, Y., et al. (2019). “Deviation-Sparse fuzzy c-means with neighbor information constraint,” IEEE Transactions on Fuzzy Systems 27(1): 185-199. [code]
• [CNIS] Mourchid, Y., et al. (2019). "A general framework for complex network-based image segmentation." Multimedia Tools and Applications 78:20191–20216.
• [MMGR-AFCF] Lei, T., et al. (2020). "Automatic Fuzzy Clustering Framework for Image Segmentation." IEEE Transactions on Fuzzy Systems 28(9): 2078-2092. [code]
• [RSFFCA] Jia, X., et al. (2020). “Robust Self-Sparse Fuzzy Clustering for Image Segmentation,” IEEE Access. [code]
• [HS] Wu, C., et al. (2020). "Superpixel Based Hierarchical Segmentation for Color Image." IEICE Transactions on Information and Systems E103.D(10): 2246-2249.
• [MSFCM] Zhou, S., et al. (2021). “A new membership scaling fuzzy c-means clustering algorithm,” IEEE Transactions on Fuzzy Systems 29(9): 2810 - 2818.
• [MME-IFODPSO] Chakraborty, R., et al. (2021). “IFODPSO‑based multi‑level image segmentation scheme aided with Masi entropy,” Journal of Ambient Intelligence and Humanized Computing
• [Inpainting] Bhugra, S., et al. (2021). "Unsupervised Learning of Affinity for Image Segmentation: An Inpainting based Approach." International Conference on Image and Vision Computing New Zealand (IVCNZ).
• [ASCQPHGS] Elaziz, M.A., et al. (2021). “Automatic Superpixel-Based Clustering for Color Image Segmentation Using q-Generalized Pareto Distribution under Linear Normalization and Hunger Games Search,” Mathematics 9: 2383.
• [LG(2)D] Wang, T., et al. (2021). "Label group diffusion for image and image pair segmentation," Pattern Recognition 112:1-11.
• [AF-GRAPH] Zhang, Y., et al. (2021). "Affinity fusion graph-based framework for natural image segmentation," IEEE Transactions on Multimedia. [code]
• [FRSC] Lei, T., et al. (2021). "Fuzzy Student’s T-Distribution Model Based on Richer Spatial Combination,” IEEE Transactions on Fuzzy Systems.
• [DTNP] Cai, T., et al. (2022). "An unsupervised segmentation method based on dynamic threshold neural P systems for color images," Information Sciences 587: 473–484.
• [SNCMWG] Ji, B., et al. (2022). "An effective color image segmentation approach using superpixel- neutrosophic C-means clustering and gradient-structural similarity," Optik
• [SCH-TRM] Lee, H., et al. (2022). "Spatial color histogram-based image segmentation using texture-aware region merging," Multimedia Tools and Applications

Citing

If you find this repository useful in your research, please consider citing:

@INPROCEEDINGS{AASP-Graph,  
  author={Y. {Zhang} and H. {Zhang} and Y. {Guo} and K. {Lin} and J. {He}},  
  booktitle={IEEE International Conference on Multimedia and Expo (ICME)},   
  title={An Adaptive Affinity Graph with Subspace Pursuit for Natural Image Segmentation},   
  year={2019}，  
  pages={802-807},}

@ARTICLE{AF-Graph,  
  author={Y. {Zhang} and M. {Liu} and J. {He} and F. {Pan} and Y. {Guo}},  
  booktitle={IEEE Transactions on Multimedia},   
  title={Affinity Fusion Graph-based Framework for Natural Image Segmentation},   
  year={2021}}

Contact & Feedback

If you have any suggestions about this project, feel free to contact me.

• [e-mail: yzhangcst[at]gmail.com]