zhengziqiang / ReshapeGAN

Introduction

ReshapeGAN is a Tensorflow-based framework for training and testing of our paper ReshapeGAN: Object Reshaping by Providing A Single Reference Image.

Note

We updated this version and achieve the high-resolution image-to-image translation on CelebA-HQ and FFHQ dataset for both 1024*1024 and 512*512. Besides, we also add the triplet loss to better preserve the identity information. And we will release the pre-trained model soon. here are some visual results.

Installation

1. We use Miniconda3 as the basic environment. If you have installed the Miniconda3 in path Conda_Path, please install tensorflow-gpu using the command Conda_Path/bin/conda install -c anaconda tensorflow-gpu==1.8.
2. Install dependencies by Conda_Path/bin/pip install -r requirements.txt (if necessary). The requirements.txt file is provided in this package.

Train

The training code will be released soon!

Test

CelebA

We provide the pre-trained model and 1000 testing images for evaluation. Pre-trained model celeba.zip (Google Drive and Baidu Drive). Testing images used in our paper: celeba_images.zip.

After download the pre-trained model and testing images, run

mkdir checkpoint
mv celeba.zip checkpoint/
cd checkpoint/
unzip celeba.zip
cd ..
mkdir -p data/celeba
mv celeba_images.zip data/celeba/
cd data/celeba
unzip celeba_images.zip
cd ../..

Then we provide two files: test_single.py and test_multiple.py for generating only one image and multiple images respectively. For generating one image:

Conda_path/bin/python test_single.py # generating one image at once
--input_img ./data/celeba/celeba_images/0000.jpg # input image, you could also change this to any input image (0001.jpg, 0002.jpg and etc)
--ref_img ./data/celeba/celeba_images/0001.jpg # reference image, you could also change this to any reference image (0001.jpg, 0002.jpg and etc)
--checkpoint_dir ./checkpoint/celeba # checkpoint path
--result_img ./demo_celeba.jpg # result image

For generating multiple images using different reference images:

Conda_path/bin/python generate_multiple.py # generate combinations of reference images, one combination has 5 different reference images, and we randomly use one image as input image.
--data_path ./data/celeba/celeba_images # images path for providing reference images
--save_path ./data/celeba/combination # path to save image combinations
--n_ref # how many reference images in one image combination, we set default is 5

Conda_path/bin/python test_multiple.py 
--input_dir ./data/celeba/combination # input image path
--checkpoint_dir ./checkpoint/celeba # checkpoint path
--result_dir ./result/celeba  # result path

To make it easier to show the synthesized images, we also provide the merge_result.py to merge the generated images to a larger image.

Conda_path/bin/python merge_results.py
--data_path ./result/celeba # the results path
--save_path ./result/celeba/merge # the path to save merged images. 

We merge 96 synthesized images to a large image, and the middle enlarged image is the input image.

UTKFace

We provide the pre-trained model and 1000 testing images for evaluation. Pre-trained model utk.zip (Google Drive and Baidu Drive). Testing images used in our paper: utk_images.zip.

After download the pre-trained model and testing images, run

mv utk.zip checkpoint/
cd checkpoint/
unzip utk.zip
cd ..
mkdir data/utk
mv utk_images.zip data/utk/
cd data/utk
unzip utk_images.zip
cd ../..

Similarly, for generating one image:

Conda_path/bin/python test_single.py # generating one image at once
--input_img ./data/utk/utk_images/0000.jpg # input image, you could also change this to any input image (0001.jpg, 0002.jpg and etc)
--ref_img ./data/utk/utk_images/0001.jpg # reference image, you could also change this to any reference image (0001.jpg, 0002.jpg and etc)
--checkpoint_dir ./checkpoint/utk # checkpoint path
--result_img ./demo_utk.jpg # result image

For generating multiple images using different reference images:

Conda_path/bin/python generate_multiple.py # generate combinations of reference images, one combination has 5 different reference images, and we randomly use one image as input image.
--data_path ./data/utk/utk_images # images path for providing reference images
--save_path ./data/utk/combination # path to save image combinations
--n_ref # how many reference images in one image combination, we set default is 5

Conda_path/bin/python test_multiple.py 
--input_dir ./data/utk/combination # input image path
--checkpoint_dir ./checkpoint/utk # checkpoint path
--result_dir ./result/utk  # result path

To make it easier to show the synthesized images, we also provide the merge_result.py to merge the generated images to a larger image.

Conda_path/bin/python merge_results.py
--data_path ./result/utk # the results path
--save_path ./result/utk/merge # the path to save merged images. 

We merge 96 synthesized images to a large image, and the middle enlarged image is the input image.

Data preparation

If you want to test your own images or train your model, you can use extract_landmark.py to get facial landmark of face images. We use dlib to obtain the 68 facial landmarks. You should follow the instructions to install dlib.

python extract_landmark.py # the python path should be the path you installed dlib
--data_path # the real RGB path
--save_path # path to save landmark images
--model_path # the predictor path mentioned in dlib, default is shape_predictor_68_face_landmarks.dat

After extracting the landmarks, we also provide a post-processing code facial_normalized.py to get center cropped image.

For the skeleton or pose extraction, we use Openpose to obtain skeleton image. Or you can use Densepose, which maybe helps a better performance for our task.

Datasets

Reshaping datasets:

• KDEF
• RaFD
• FEI
• CelebA
• UTKFace
• Yale
• WSEFEP
• ADFES
• IIIT-CFW
• PHOTO-SKETCH
• CUHK
• Face Sketch database

Data preparation

Reshaping by within-domain guidance with paired data

├── demo
   ├── train
       ├── 000001.jpg 
       ├── 000002.jpg
       └── ...
   ├── test
       ├── a.jpg (The test image that you want)
       ├── b.png
       └── ... 

Reshaping by within-domain guidance with unpaired data

├── demo
   ├── train
       ├── 000001.jpg 
       ├── 000002.jpg
       └── ...
   ├── test
       ├── a.jpg (The test image that you want)
       ├── b.png
       └── ... 

Reshaping by cross-domain guidance with unparied data

├── demo
   ├── train
       ├── 000001.jpg 
       ├── 000002.jpg
       └── ...
   ├── test
       ├── a.jpg (The test image that you want)
       ├── b.png
       └── ...
   ├── attribute.txt (For domain attribute information for training)
   ├── attribute_test.txt (For domain attribute information for training) 

Losses

• DRAGAN: Perturbed loss.
• LSGAN: Least Square GAN.

Tutorial

Train

Codes will be released soon!

ReshapeGAN settings

The detailed ReshapeGAN model

Reshaping by within-domain guidance with paired data

Reshaping by within-domain guidance with unpaired data

Reshaping by cross-domain guidance with unpaired data

Results

CelebA

The 5 × 5 outputs by our ReshapeGAN using random 5 images as both inputs and references on CelebA dataset.

Random 96 synthesized images by our ReshapeGAN from one input sample on CelebA dataset. The middle enlarged image shows the input.

UTK

Random 96 generated results by our ReshapeGAN from one input sample on UTKFace dataset. The middle enlarged image shows the input.

cat head

Random 96 synthesized images by our ReshapeGAN from one input sample on Cat head dataset. The middle enlarged image shows the input.

Panama

Random 96 synthesized images by our ReshapeGAN from one input sample on Panama dataset. The middle enlarged image shows the input.