Domain Aware Universal Style Transfer

Official Pytorch Implementation of 'Domain Aware Universal Style Transfer' (ICCV 2021)

Domain Aware Universal Style Transfer

Kibeom Hong (Yonsei Univ.), Seogkyu Jeon (Yonsei Univ.), Huan Yang (Microsoft Research), Jianlong Fu (Microsoft Research), Hyeran Byun (Yonsei Univ.)

Paper : https://openaccess.thecvf.com/content/ICCV2021/html/Hong_Domain-Aware_Universal_Style_Transfer_ICCV_2021_paper.html

Paper[Arxiv] : https://arxiv.org/abs/2108.04441

Abstract: Style transfer aims to reproduce content images with the styles from reference images. Existing universal style transfer methods successfully deliver arbitrary styles to original images either in an artistic or a photo-realistic way. However, the range of “arbitrary style” defined by existing works is bounded in the particular domain due to their structural limitation. Specifically, the degrees of content preservation and stylization are established according to a predefined target domain. As a result, both photo-realistic and artistic models have difficulty in performing the desired style transfer for the other domain. To overcome this limitation, we propose a unified architecture, Domain-aware Style Transfer Networks (DSTN) that transfer not only the style but also the property of domain (i.e., domainness) from a given reference image. To this end, we design a novel domainness indicator that captures the domainness value from the texture and structural features of reference images. Moreover, we introduce a unified framework with domain-aware skip connection to adaptively transfer the stroke and palette to the input contents guided by the domainness indicator. Our extensive experiments validate that our model produces better qualitative results and outperforms previous methods in terms of proxy metrics on both artistic and photo-realistic stylizations.

Notice

Due to the policy change of google-drive which is saving pre-trained weights, it is no longer available on google drive! If you need pre-trained wieght, please contact us via personal email([email protected]) and we will send it to you right away. Thank you :)

Prerequisites

Dependency

• Python 3.6
• CUDA 11.1
• Pytorch 1.7
• Check the requirements.txt

pip install -r requirements.txt

Usage

Set pretrained weights

• Pretrained models for encoder(VGG-19) can be found in the ./baseline_checkpoints

• Prepare pretrained models for Domainnes Indicator

  • Domainnes Indicator can be downloaded at style_indicator.pth

• Prepare pretrained models for Decoder

  • Vanilla version can be downloaded at Decoder.pth
  • Adversarial version can be downloaded at Decoder_adversarial.pth

• Move these pretrained weights to each folders:

  • style_indicator.pth -> ./train_results/StyleIndicator/log/
  • decoder.pth -> ./train_results/Decoder/log/
  • decoder_adversarial.pth -> ./train_results/Decoder_adversarial/log/

  (Please rename decoder_adversarial.pth -> decoder.pth)

Inference (Automatic)

• Vanilla decoder

bash scripts/transfer.sh

• Decoder with adversarial loss

bash scripts/transfer_adversarial.sh

Inference (User Guided)

• Vanilla decoder (You should set --alpha value in script file)

bash scripts/transfer_user_guided.sh

• Decoder with adversarial loss (You should set --alpha value in script file)

bash scripts/transfer_adversarial_user_guided.sh

Inference (Interpolation)

bash scripts/interpolate.sh

Training

Our networks could be trained with end-to-end manner. However, we recommend to train StyleIndicator and Decoder respectively.

• (1 step) Train StyleIndicator

bash scripts/train_indicator.sh

• (2 step) Train Decoder

bash scripts/train_decoder.sh

Evaluation

Available soon

Citation

If you find this work useful for your research, please cite:

@InProceedings{Hong_2021_ICCV,
    author    = {Hong, Kibeom and Jeon, Seogkyu and Yang, Huan and Fu, Jianlong and Byun, Hyeran},
    title     = {Domain-Aware Universal Style Transfer},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2021},
    pages     = {14609-14617}
}

@article{Hong2021DomainAwareUS,
  title={Domain-Aware Universal Style Transfer},
  author={Kibeom Hong and Seogkyu Jeon and Huan Yang and Jianlong Fu and H. Byun},
  journal={ArXiv},
  year={2021},
  volume={abs/2108.04441}
}

Contact

If you have any question or comment, please contact the first author of this paper - Kibeom Hong

[email protected]