MaskRCNN-Modanet-Fashion-Segmentation-and-Classification

With the contributions of Sergen and Murat.

The project aims to classify the clothes in terms of its type (shirt, tshirt, pant etc.) and season (summer, winter etc.) simultaneously. Initially, image segmentation was applied to images to get clothes types and their boundaries. Then, the annotations of clothes were generated from boundaries and fed into the image classification model. Concequently, images were classified as skirt_summer, pant_spring, tshirt_autumn etc.

There are three main subprocess exist as given below:

1. Training Model-1 from Modanet Dataset and Generate Custom Dataset's Labels
2. Training Model-2 from Custom Dataset
3. Evaluate test results

For training the model, it is needed to labeled images but our custom dataset is not labeled. There are so many image to label and we do not want to waste time for it. The our idea which can handle this issue is use prelabeled fashion dataset to train a model. And then, our custom dataset is given as input into produced model and got segmentation of clothes as result. From this segmentation result's mask, the labels are produces for each clothes in each images.

In this way, the labels are produced by automatically from using model1. After that, our custom dataset's labels is ready to train for classification of season.

1. Training Model1 from Modanet Dataset and Generate Custom Dataset's Labels

Prepare Modanet Dataset

Modanet dataset has over 40GB fashion clothes images and its annotations. While downloading, it is possible to choice which 40GB or 2GB dataset. In order to more manageable, we prefer 2GB dataset. The annotation data format of ModaNet follows the same style as COCO-dataset.

We suggest that use google colab to download dataset. There is our Modanet-Download-Colab-File.

After download, there are over 10.000 images and json file which is include their labels. We parsed dataset into %80 instances_train.json, %10 instances_test.json and %10 instances_val.json. It is remember that the json file is also need to seperate as given percentages.

But there is an issue that about annotations type. The modanet is provide us annotations with coco format but we used Vgg-annotation type. Therefore, we converted parsed coco json files to vgg json files and located them into train and validation folder as namely via_region_data.json.

As a result, we have given annotations format:

{
 "377": {
  "filename": "0000377.jpg",
  "size": -1,
  "regions": [
   {
    "shape_attributes": {
     "name": "polygon",
     "all_points_x": [
      244,
      247,
      248 ...
     ],
     "all_points_y": [
      234,
      257,
      271 ...
     ]
    },
    "region_attributes": {
     "fashion": "1"
    }
   }
  ],
  "file_attributes": {
   "width": 400,
   "height": 600
  }
 }
}

The dataset should be located into dataset folder with its annotations json file seperately as train and validation. The test images are located into images folder which is located into base folder.

Labels

Each polygon (bounding box, segmentation mask) annotation is assigned to one of the following labels:

Train Model-1 using Modanet Dataset

Our custom training and test files is located into given folder as link. You can start to training without any pretrained model as give model parameter as --model=coco . In this way, the coco weighted model will downloaded automatically. In other option is use own model.

You can run command directly from the command line as such:

# Train a new model starting from pre-trained COCO weights
python3 model1/samples/custom/trainModaNet.py train --dataset=/path/to/coco/ --model=coco

# Continue training a model that you had trained earlier
python3 model1/samples/custom/trainModaNet.py train --dataset=/path/to/coco/ --model=/path/to/weights.h5

# For our model
python3 model1/samples/custom/trainModaNet.py train --dataset=/path/to/coco/ --model=mask_rcnn_Modanet.h5

You can also run the evaluation code with:

# Run evaluation code on the last trained model
python3 testModaNet.py

Segmentation Results

Sample segmentation result is shown as below:

Generate Custom Dataset's Labels

Our custom dataset was passing into trained model mask_rcnn_Modanet.h5 as input. As a result, we got segmentation of clothes. The segmentation result has mask of segmented clothes and their class ID. Using this mask we produced detected clothes's boundaries and their labels as vgg annotation type. This is how we can handle the issue which is already talked above as our unlabel dataset.

The generateVggAnno.py file is used for generate vgg annotations from our custom dataset. It is used mask_rcnn_Modanet.h5 model for images segmentation and labelling them.

# Run evaluation code on the last trained model
python3 generateVggAnno.py

After the run generateVggAnno.py, the custom_dataset_annotations.json is produced as a result on the same folder. We use this annotations file for training model-2.

2. Training Model-2 from Custom Dataset

Prepare Custom Dataset

After the annotations are generated from previous step as custom_dataset_annotations.json, the json file and also dataset is parsed into %80 train, %10 validation and test as given percentage. The annotations files for train and validation images namely via_region_data.json are also located into same folder with train and validation images under dataset folder. The test images are located into images folder which is located into base folder.

Labels

Our custom dataset has normally 10 different fashion and its season labels. We can get class name and season label from images file name. In the generateVggAnno.py, these file names are used for generate annotations. Actually each class has 5 subclass which specified seasons as summer, winter, autumn, spring and all. Because of unsufficient number of sample, we had to decrease class label and just use the most exist ones.

Each polygon (bounding box, segmentation mask) annotation is assigned to one of the following labels:

usage: fashionClassName_category

Categories: yaz->summer, sonb->autumn, ilkb->spring, kis->winter, 4mev->all

Train Model-2 using Custom Dataset

Our custom training and test files is located into given folder as link. You can start to training without any pretrained model as give model parameter as --model=coco . In this way, the coco weighted model will downloaded automatically. In other option is use own model. But we prefer use our pretrained model produced from model-1 --model=mask_rcnn_Modanet.h5-rcnn-

You can run command directly from the command line as such:

# Train a new model starting from pre-trained COCO weights
python3 model2/samples/custom/trainSeasonData.py train --dataset=/path/to/coco/ --model=coco

# Continue training a model that you had trained earlier
python3 model2/samples/custom/trainSeasonData.py train --dataset=/path/to/coco/ --model=/path/to/weights.h5

# For our model
python3 model2/samples/custom/trainSeasonData.py train --dataset=/path/to/coco/ --model=mask_rcnn_Modanet.h5

3. Evaluate test results

After the training of model-2 from custom dataset, the mask_rcnn_Modanet.h5 is produced as result. And now using this model, we can get seasons of clothes from given images as input.

You can run the evaluation code with:

# Run evaluation code on the last trained model
python3 testSeason.py

Compare Model-1 and Model-2 Results

Model-1 Outputs	Model-2 Outputs