GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → PeterouZh → CIPS-3D

PeterouZh / CIPS-3D

Licence: MIT license
3D-aware GANs based on NeRF (arXiv).

Programming Languages

python
139335 projects - #7 most used programming language
shell
77523 projects

Labels

pytorchgenerative-adversarial-networknerf

Projects that are alternatives of or similar to CIPS-3D

Paper-Notes
Paper notes in deep learning/machine learning and computer vision
Stars: ✭ 37 (-93.42%)
Mutual labels:  generative-adversarial-network
progressive-growing-of-gans.pytorch
Unofficial PyTorch implementation of "Progressive Growing of GANs for Improved Quality, Stability, and Variation".
Stars: ✭ 51 (-90.93%)
Mutual labels:  generative-adversarial-network
MAD-GAN-MLCAMP
Repository for MAD-GAN Paper done in ML CAMP Jeju
Stars: ✭ 17 (-96.98%)
Mutual labels:  generative-adversarial-network
GAN-LTH
[ICLR 2021] "GANs Can Play Lottery Too" by Xuxi Chen, Zhenyu Zhang, Yongduo Sui, Tianlong Chen
Stars: ✭ 24 (-95.73%)
Mutual labels:  generative-adversarial-network
SMILE
SMILE: Semantically-guided Multi-attribute Image and Layout Editing, ICCV Workshops 2021.
Stars: ✭ 28 (-95.02%)
Mutual labels:  generative-adversarial-network
Deep-Learning-Pytorch
A repo containing code covering various aspects of deep learning on Pytorch. Great for beginners and intermediate in the field
Stars: ✭ 59 (-89.5%)
Mutual labels:  generative-adversarial-network
HashGAN
HashGAN: Deep Learning to Hash with Pair Conditional Wasserstein GAN
Stars: ✭ 63 (-88.79%)
Mutual labels:  generative-adversarial-network
Deep-Fakes
No description or website provided.
Stars: ✭ 88 (-84.34%)
Mutual labels:  generative-adversarial-network
DCGAN-CIFAR10
A implementation of DCGAN (Deep Convolutional Generative Adversarial Networks) for CIFAR10 image
Stars: ✭ 18 (-96.8%)
Mutual labels:  generative-adversarial-network
timegan-pytorch
This repository is a non-official implementation of TimeGAN (Yoon et al., NIPS2019) using PyTorch.
Stars: ✭ 46 (-91.81%)
Mutual labels:  generative-adversarial-network
Pytorch-conditional-GANs
Implementation of Conditional Generative Adversarial Networks in PyTorch
Stars: ✭ 91 (-83.81%)
Mutual labels:  generative-adversarial-network
Self-Supervised-GANs
Tensorflow Implementation for paper "self-supervised generative adversarial networks"
Stars: ✭ 34 (-93.95%)
Mutual labels:  generative-adversarial-network
MMD-GAN
Improving MMD-GAN training with repulsive loss function
Stars: ✭ 82 (-85.41%)
Mutual labels:  generative-adversarial-network
gans-2.0
Generative Adversarial Networks in TensorFlow 2.0
Stars: ✭ 76 (-86.48%)
Mutual labels:  generative-adversarial-network
Pytorch models
PyTorch study
Stars: ✭ 14 (-97.51%)
Mutual labels:  generative-adversarial-network
speech-enhancement-WGAN
speech enhancement GAN on waveform/log-power-spectrum data using Improved WGAN
Stars: ✭ 35 (-93.77%)
Mutual labels:  generative-adversarial-network
text2imageNet
Generate image from text with Generative Adversarial Network
Stars: ✭ 26 (-95.37%)
Mutual labels:  generative-adversarial-network
bmusegan
Code for “Convolutional Generative Adversarial Networks with Binary Neurons for Polyphonic Music Generation”
Stars: ✭ 58 (-89.68%)
Mutual labels:  generative-adversarial-network
NeuRay
[CVPR2022] Neural Rays for Occlusion-aware Image-based Rendering
Stars: ✭ 291 (-48.22%)
Mutual labels:  nerf
text2image-benchmark
Performance comparison of existing GAN based Text To Image algorithms. (GAN-CLS, StackGAN, TAC-GAN)
Stars: ✭ 25 (-95.55%)
Mutual labels:  generative-adversarial-network
View All Similar Projects ➔

CIPS-3D

This repository contains the code of the paper,
CIPS-3D: A 3D-Aware Generator of GANs Based on Conditionally-Independent Pixel Synthesis.

✔️ (2022-1-5) The code has been refactored. Please refer to the scripts in exp/cips3d/bash. Please upgrade the tl2 package with pip install -I tl2.

✔️ (2021-11-26) The configuration files (yaml files) for training are being released.

✔️ (2021-10-27) All the code files have been released. The configuration files (yaml files) for training will be released next. Now I have provided a GUI script and models to facilitate the experiment of network interpolation (see below). If you find any problems, please open an issue. Have fun with it.

✔️ (2021-10-25) Thank you for your kind attention. The github star has reached two hundred. I will open source the training code in the near future.

✔️ (2021-10-20) We are planning to publish the training code here in December. But if the github star reaches two hundred, I will advance the date. Stay tuned 🕙.

Demo videos

demo1.mp4
demo2.mp4
demo_animal_finetuned.mp4
demo3.mp4
demo4.mp4
demo5.mp4
Mirror symmetry problem

The problem of mirror symmetry refers to the sudden change of the direction of the bangs near the yaw angle of pi/2. We propose to use an auxiliary discriminator to solve this problem (please see the paper).

Note that in the initial stage of training, the auxiliary discriminator must dominate the generator more than the main discriminator does. Otherwise, if the main discriminator dominates the generator, the mirror symmetry problem will still occur. In practice, progressive training is able to guarantee this. We have trained many times from scratch. Adding an auxiliary discriminator stably solves the mirror symmetry problem. If you find any problems with this idea, please open an issue.

Prepare environment 
git clone --recursive https://github.com/PeterouZh/CIPS-3D.git
cd CIPS-3D

# Create virtual environment
conda create -y --name cips3d python=3.6.7
conda activate cips3d

pip install torch==1.8.2+cu102 torchvision==0.9.2+cu102 -f https://download.pytorch.org/whl/lts/1.8/torch_lts.html

pip install --no-cache-dir -r requirements.txt
pip install -I tl2

pip install -e torch_fidelity_lib
pip install -e pytorch_ema_lib

Model interpolation (web demo)

Download the pre-trained checkpoints.

Execute this command:

streamlit run --server.port 8650 -- scripts/web_demo.py  \
  --outdir results/model_interpolation \
  --cfg_file configs/web_demo.yaml \
  --command model_interpolation

Then open the browser: http://your_ip_address:8650.

You can debug this script with this command:

python scripts/web_demo.py  \
  --outdir results/model_interpolation \
  --cfg_file configs/web_demo.yaml \
  --command model_interpolation \
  --debug True

Pre-trained checkpoints

ffhq_exp
FFHQ_r256 train_ffhq_high-20220105_143314_190
AFHQ_r256 finetune_afhq-20220124_193407_473
CartoonFaces_r256 finetune_photo2cartoon-20220107_172255_454
Prepare dataset

FFHQ: Download FFHQ dataset images1024x1024 (89.1 GB)

# Downsampling images in advance to speed up training
python scripts/dataset_tool.py \
    --source=datasets/ffhq/images1024x1024 \
    --dest=datasets/ffhq/downsample_ffhq_256x256.zip \
    --width=256 --height=256

CartoonFaces Download photo2cartoon dataset

# Prepare training dataset.
python scripts/dataset_tool.py \
    --source=datasets/photo2cartoon/photo2cartoon \
    --dest=datasets/photo2cartoon/photo2cartoon_stylegan2.zip

AFHQ Download afhq dataset

# Prepare training dataset.
python scripts/dataset_tool.py \
    --source=datasets/AFHQv2/AFHQv2 \
    --dest=datasets/AFHQv2/AFHQv2_stylegan2.zip

Training

Please refer to the scripts in exp/cips3d/bash. I will release all the pre-trained models when the reproducing is over.

running order:

  • exp/cips3d/bash/ffhq_exp:

    • train_ffhq_r32.sh -> train_ffhq_r64.sh -> train_ffhq_r128.sh -> train_ffhq_r256.sh
    • eval_fid.sh

  • exp/cips3d/bash/finetuning_exp: (require pre-trained models from the above step)

    • finetune_photo2cartoon.sh

developing:

  • exp/cips3d/bash/ffhq_exp_v1:

  • exp/cips3d/bash/afhq_exp:

Bug fixed

  • If the training process is blocked when training with multi GPUs, please upgrade the tl2 via pip install -I tl2
Old readme

Note:

  • In order to ensure that this code is consistent with my original dirty code, please follow me to reproduce the results using this code step by step.
  • The training script train_v16.py is dirty, but I'm not going to refactor it. After all, it still works stably.

Start training at 64x64

Training:

export CUDA_HOME=/usr/local/cuda-10.2/
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export PYTHONPATH=.
python exp/dev/nerf_inr/scripts/train_v16.py \
    --port 8888 \
    --tl_config_file configs/train_ffhq.yaml \
    --tl_command train_ffhq \
    --tl_outdir results/train_ffhq \
    --tl_opts curriculum.new_attrs.image_list_file datasets/ffhq/images256x256_image_list.txt \
      D_first_layer_warmup True

Dummy training (for debug):

export CUDA_HOME=/usr/local/cuda-10.2/
export CUDA_VISIBLE_DEVICES=1
python exp/dev/nerf_inr/scripts/train_v16.py \
    --port 8888 \ 
    --tl_config_file configs/train_ffhq.yaml \
    --tl_command train_ffhq \
    --tl_outdir results/train_ffhq_debug \ 
    --tl_debug \
    --tl_opts curriculum.new_attrs.image_list_file datasets/ffhq/images256x256_image_list.txt \
      num_workers 0 num_images_real_eval 10 num_images_gen_eval 2

When the FID of the 64x64 model reaches about 16, we start the next step: resume training at 128x128. Let's wait for the training (about 2 days or less).

Reproduced results: best_FID=15.27

Resume training at 128x128 from the 64x64 models

Training:

export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export PYTHONPATH=.
python exp/dev/nerf_inr/scripts/train_v16.py \
    --port 8888 \
    --tl_config_file configs/train_ffhq.yaml \
    --tl_command train_ffhq_r128 \
    --tl_outdir results/train_ffhq \
    --tl_resume \
    --tl_resumedir results/train_ffhq \
    --tl_opts curriculum.new_attrs.image_list_file datasets/ffhq/images256x256_image_list.txt \
      D_first_layer_warmup True reset_best_fid True update_aux_every 16 d_reg_every 1 train_aux_img True

Dummy training (for debug):

export CUDA_HOME=/usr/local/cuda-10.2/
export CUDA_VISIBLE_DEVICES=1
python exp/dev/nerf_inr/scripts/train_v16.py \
    --port 8888 \ 
    --tl_config_file configs/train_ffhq.yaml \
    --tl_command train_ffhq_r128 \
    --tl_outdir results/train_ffhq \ 
    --tl_resume \
    --tl_resumedir results/train_ffhq \
    --tl_debug \
    --tl_opts curriculum.new_attrs.image_list_file datasets/ffhq/images256x256_image_list.txt \
      num_workers 0 num_images_real_eval 10 num_images_gen_eval 2 reset_best_fid True

When the FID of the 128x128 model reaches about 16, we start the next step.

Some hyperparameters may be different from the original experiment. Hope it works normally. Let's wait for the training (maybe longer).

Resume training at 256x256 from the 128x128 models



Finetune INR Net






Citation


If you find our work useful in your research, please cite:



@article{zhou2021CIPS3D,
  title = {{{CIPS}}-{{3D}}: A {{3D}}-{{Aware Generator}} of {{GANs Based}} on {{Conditionally}}-{{Independent Pixel Synthesis}}},
  shorttitle = {{{CIPS}}-{{3D}}},
  author = {Zhou, Peng and Xie, Lingxi and Ni, Bingbing and Tian, Qi},
  year = {2021},
  eprint = {2110.09788},
  eprinttype = {arxiv},
}




Acknowledgments




          
Note that the project description data, including the texts, logos, images, and/or trademarks, 
for each open source project belongs to its rightful owner. 
If you wish to add or remove any projects, please contact us at [email protected].