GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → paarthneekhara → Text To Image

paarthneekhara / Text To Image

Licence: mit
Text to image synthesis using thought vectors

Programming Languages

python
139335 projects - #7 most used programming language

Labels

deep-learningtensorflowgenerative-adversarial-networkskip-thought-vectors

Projects that are alternatives of or similar to Text To Image

skip-thought-gan
Generating Text through Adversarial Training(GAN) using Skip-Thought Vectors
Stars: ✭ 44 (-97.86%)
Mutual labels:  generative-adversarial-network, skip-thought-vectors
Accel Brain Code
The purpose of this repository is to make prototypes as case study in the context of proof of concept(PoC) and research and development(R&D) that I have written in my website. The main research topics are Auto-Encoders in relation to the representation learning, the statistical machine learning for energy-based models, adversarial generation networks(GANs), Deep Reinforcement Learning such as Deep Q-Networks, semi-supervised learning, and neural network language model for natural language processing.
Stars: ✭ 166 (-91.91%)
Mutual labels:  generative-adversarial-network
Stylegan.pytorch
A PyTorch implementation for StyleGAN with full features.
Stars: ✭ 150 (-92.69%)
Mutual labels:  generative-adversarial-network
A Pytorch Tutorial To Super Resolution
Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network | a PyTorch Tutorial to Super-Resolution
Stars: ✭ 157 (-92.35%)
Mutual labels:  generative-adversarial-network
Netgan
Implementation of the paper "NetGAN: Generating Graphs via Random Walks".
Stars: ✭ 152 (-92.59%)
Mutual labels:  generative-adversarial-network
Anime Face Gan Keras
A DCGAN to generate anime faces using custom mined dataset
Stars: ✭ 161 (-92.15%)
Mutual labels:  generative-adversarial-network
Tensorflow Infogan
🎎 InfoGAN: Interpretable Representation Learning
Stars: ✭ 149 (-92.74%)
Mutual labels:  generative-adversarial-network
Image generator
DCGAN image generator 🖼️.
Stars: ✭ 173 (-91.57%)
Mutual labels:  generative-adversarial-network
Dcgan wgan wgan Gp lsgan sngan rsgan began acgan pggan tensorflow
Implementation of some different variants of GANs by tensorflow, Train the GAN in Google Cloud Colab, DCGAN, WGAN, WGAN-GP, LSGAN, SNGAN, RSGAN, RaSGAN, BEGAN, ACGAN, PGGAN, pix2pix, BigGAN
Stars: ✭ 166 (-91.91%)
Mutual labels:  generative-adversarial-network
Stylegan2 Pytorch
Simplest working implementation of Stylegan2, state of the art generative adversarial network, in Pytorch. Enabling everyone to experience disentanglement
Stars: ✭ 2,656 (+29.43%)
Mutual labels:  generative-adversarial-network
Mmediting
OpenMMLab Image and Video Editing Toolbox
Stars: ✭ 2,618 (+27.58%)
Mutual labels:  generative-adversarial-network
Progressive growing of gans tensorflow
Tensorflow implementation of PROGRESSIVE GROWING OF GANS FOR IMPROVED QUALITY, STABILITY, AND VARIATION
Stars: ✭ 152 (-92.59%)
Mutual labels:  generative-adversarial-network
Mmd Gan
MMD-GAN: Towards Deeper Understanding of Moment Matching Network
Stars: ✭ 161 (-92.15%)
Mutual labels:  generative-adversarial-network
Isketchnfill
Software that can autocomplete sketches as the user starts drawing.
Stars: ✭ 151 (-92.64%)
Mutual labels:  generative-adversarial-network
Gannotation
GANnotation (PyTorch): Landmark-guided face to face synthesis using GANs (And a triple consistency loss!)
Stars: ✭ 167 (-91.86%)
Mutual labels:  generative-adversarial-network
Dvbpr
Personalized Fashion Recommendation and Generation
Stars: ✭ 150 (-92.69%)
Mutual labels:  generative-adversarial-network
Anogan Keras
Unsupervised anomaly detection with generative model, keras implementation
Stars: ✭ 157 (-92.35%)
Mutual labels:  generative-adversarial-network
Frontalization
Pytorch deep learning face frontalization model
Stars: ✭ 160 (-92.2%)
Mutual labels:  generative-adversarial-network
Tgan
Generative adversarial training for generating synthetic tabular data.
Stars: ✭ 173 (-91.57%)
Mutual labels:  generative-adversarial-network
Edge Connect
EdgeConnect: Structure Guided Image Inpainting using Edge Prediction, ICCV 2019 https://arxiv.org/abs/1901.00212
Stars: ✭ 2,163 (+5.41%)
Mutual labels:  generative-adversarial-network
View All Similar Projects ➔

Text To Image Synthesis Using Thought Vectors

Join the chat at https://gitter.im/text-to-image/Lobby

This is an experimental tensorflow implementation of synthesizing images from captions using Skip Thought Vectors. The images are synthesized using the GAN-CLS Algorithm from the paper Generative Adversarial Text-to-Image Synthesis. This implementation is built on top of the excellent DCGAN in Tensorflow. The following is the model architecture. The blue bars represent the Skip Thought Vectors for the captions.

Model architecture

Image Source : Generative Adversarial Text-to-Image Synthesis Paper

Requirements

Datasets

  • All the steps below for downloading the datasets and models can be performed automatically by running python download_datasets.py. Several gigabytes of files will be downloaded and extracted.
  • The model is currently trained on the flowers dataset. Download the images from this link and save them in Data/flowers/jpg. Also download the captions from this link. Extract the archive, copy the text_c10 folder and paste it in Data/flowers.
  • Download the pretrained models and vocabulary for skip thought vectors as per the instructions given here. Save the downloaded files in Data/skipthoughts.
  • Make empty directories in Data, Data/samples, Data/val_samples and Data/Models. They will be used for sampling the generated images and saving the trained models.

Usage

  • Data Processing : Extract the skip thought vectors for the flowers data set using :
python data_loader.py --data_set="flowers"

  • Training

    • Basic usage python train.py --data_set="flowers"
    • Options
      • z_dim: Noise Dimension. Default is 100.
      • t_dim: Text feature dimension. Default is 256.
      • batch_size: Batch Size. Default is 64.
      • image_size: Image dimension. Default is 64.
      • gf_dim: Number of conv in the first layer generator. Default is 64.
      • df_dim: Number of conv in the first layer discriminator. Default is 64.
      • gfc_dim: Dimension of gen untis for for fully connected layer. Default is 1024.
      • caption_vector_length: Length of the caption vector. Default is 1024.
      • data_dir: Data Directory. Default is Data/.
      • learning_rate: Learning Rate. Default is 0.0002.
      • beta1: Momentum for adam update. Default is 0.5.
      • epochs: Max number of epochs. Default is 600.
      • resume_model: Resume training from a pretrained model path.
      • data_set: Data Set to train on. Default is flowers.

  • Generating Images from Captions

    • Write the captions in text file, and save it as Data/sample_captions.txt. Generate the skip thought vectors for these captions using:
    python generate_thought_vectors.py --caption_file="Data/sample_captions.txt"
    • Generate the Images for the thought vectors using:
    python generate_images.py --model_path=<path to the trained model> --n_images=8

    n_images specifies the number of images to be generated per caption. The generated images will be saved in Data/val_samples/. python generate_images.py --help for more options.

Sample Images Generated

Following are the images generated by the generative model from the captions.

Caption Generated Images
the flower shown has yellow anther red pistil and bright red petals
this flower has petals that are yellow, white and purple and has dark lines
the petals on this flower are white with a yellow center
this flower has a lot of small round pink petals.
this flower is orange in color, and has petals that are ruffled and rounded.
the flower has yellow petals and the center of it is brown

Implementation Details

  • Only the uni-skip vectors from the skip thought vectors are used. I have not tried training the model with combine-skip vectors.
  • The model was trained for around 200 epochs on a GPU. This took roughly 2-3 days.
  • The images generated are 64 x 64 in dimension.
  • While processing the batches before training, the images are flipped horizontally with a probability of 0.5.
  • The train-val split is 0.75.

Pre-trained Models

  • Download the pretrained model from here and save it in Data/Models. Use this path for generating the images.

TODO

  • Train the model on the MS-COCO data set, and generate more generic images.
  • Try different embedding options for captions(other than skip thought vectors). Also try to train the caption embedding RNN along with the GAN-CLS model.

References

Alternate Implementations

License

MIT

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].