GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → Shobhit20 → Image Captioning

Shobhit20 / Image Captioning

Licence: mit
Image Captioning: Implementing the Neural Image Caption Generator with python

Programming Languages

python
139335 projects - #7 most used programming language

Labels

deep-learningmachine-learningcomputer-visionconvolutional-neural-networkslstmrecurrent-neural-networksconvolutional-networkslstm-neural-networksimage-captioning

Projects that are alternatives of or similar to Image Captioning

Image Caption Generator
[DEPRECATED] A Neural Network based generative model for captioning images using Tensorflow
Stars: ✭ 141 (+171.15%)
Mutual labels:  convolutional-neural-networks, lstm, recurrent-neural-networks, lstm-neural-networks, image-captioning
Pytorch Learners Tutorial
PyTorch tutorial for learners
Stars: ✭ 97 (+86.54%)
Mutual labels:  convolutional-neural-networks, lstm, recurrent-neural-networks, lstm-neural-networks
Image Caption Generator
A neural network to generate captions for an image using CNN and RNN with BEAM Search.
Stars: ✭ 126 (+142.31%)
Mutual labels:  convolutional-neural-networks, lstm, recurrent-neural-networks, image-captioning
Lstm anomaly thesis
Anomaly detection for temporal data using LSTMs
Stars: ✭ 178 (+242.31%)
Mutual labels:  lstm, recurrent-neural-networks, lstm-neural-networks
Bitcoin Price Prediction Using Lstm
Bitcoin price Prediction ( Time Series ) using LSTM Recurrent neural network
Stars: ✭ 67 (+28.85%)
Mutual labels:  lstm, recurrent-neural-networks, lstm-neural-networks
Deep Learning Time Series
List of papers, code and experiments using deep learning for time series forecasting
Stars: ✭ 796 (+1430.77%)
Mutual labels:  lstm, recurrent-neural-networks, lstm-neural-networks
Automatic Image Captioning
Generating Captions for images using Deep Learning
Stars: ✭ 84 (+61.54%)
Mutual labels:  convolutional-neural-networks, lstm-neural-networks, image-captioning
Pytorch Kaldi
pytorch-kaldi is a project for developing state-of-the-art DNN/RNN hybrid speech recognition systems. The DNN part is managed by pytorch, while feature extraction, label computation, and decoding are performed with the kaldi toolkit.
Stars: ✭ 2,097 (+3932.69%)
Mutual labels:  lstm, recurrent-neural-networks, lstm-neural-networks
Deepseqslam
The Official Deep Learning Framework for Route-based Place Recognition
Stars: ✭ 49 (-5.77%)
Mutual labels:  convolutional-neural-networks, lstm, recurrent-neural-networks
Awesome Tensorlayer
A curated list of dedicated resources and applications
Stars: ✭ 248 (+376.92%)
Mutual labels:  convolutional-neural-networks, recurrent-neural-networks, lstm-neural-networks
Personality Detection
Implementation of a hierarchical CNN based model to detect Big Five personality traits
Stars: ✭ 338 (+550%)
Mutual labels:  convolutional-neural-networks, lstm, lstm-neural-networks
Keras Anomaly Detection
Anomaly detection implemented in Keras
Stars: ✭ 335 (+544.23%)
Mutual labels:  convolutional-neural-networks, lstm, recurrent-neural-networks
Easy Deep Learning With Keras
Keras tutorial for beginners (using TF backend)
Stars: ✭ 367 (+605.77%)
Mutual labels:  convolutional-neural-networks, lstm, convolutional-networks
Chainer Rnn Ner
Named Entity Recognition with RNN, implemented by Chainer
Stars: ✭ 19 (-63.46%)
Mutual labels:  lstm, recurrent-neural-networks
Neural Image Captioning
Implementation of Neural Image Captioning model using Keras with Theano backend
Stars: ✭ 12 (-76.92%)
Mutual labels:  lstm, image-captioning
Tensorflow Lstm Sin
TensorFlow 1.3 experiment with LSTM (and GRU) RNNs for sine prediction
Stars: ✭ 52 (+0%)
Mutual labels:  lstm, recurrent-neural-networks
Tf cnnvis
CNN visualization tool in TensorFlow
Stars: ✭ 769 (+1378.85%)
Mutual labels:  convolutional-neural-networks, convolutional-networks
Rnn lstm gesture recog
For recognising hand gestures using RNN and LSTM... Implementation in TensorFlow
Stars: ✭ 14 (-73.08%)
Mutual labels:  recurrent-neural-networks, lstm-neural-networks
Ml In Tf
Get started with Machine Learning in TensorFlow with a selection of good reads and implemented examples!
Stars: ✭ 45 (-13.46%)
Mutual labels:  convolutional-neural-networks, recurrent-neural-networks
Machine Learning Curriculum
💻 Make machines learn so that you don't have to struggle to program them; The ultimate list
Stars: ✭ 761 (+1363.46%)
Mutual labels:  convolutional-neural-networks, recurrent-neural-networks
View All Similar Projects ➔

Image Captioning

Image captioning is describing an image fed to the model. The task of object detection has been studied for a long time but recently the task of image captioning is coming into light. This repository contains the "Neural Image Caption" model proposed by Vinyals et. al.[1]

Dataset

The dataset used is flickr8k. You can request the data here. An email for the links of the data to be downloaded will be mailed to your id. Extract the images in Flickr8K_Data and the text data in Flickr8K_Text.

Requirements

  1. Tensorflow
  2. Keras
  3. Numpy
  4. h5py
  5. Pandas
  6. Pillow
  7. Pyttsx

Steps to execute

  1. After extracting the data, execute the preprocess_data.py file by locating the file directory and execute "python preprocess_data.py". This file adds "start " and " end" token to the training and testing text data. On execution the file creates new txt files in Flickr8K_Text folder.

  2. Execute the encode_image.py file by typing "python encode_image.py" in the terminal window of the file directory. This creates image_encodings.p which generates image encodings by feeding the image to VGG16 model. In case the weights are not directly available in your temp directory, the weights will be downloaded first.

  3. Execute the train.py file in terminal window as "python train.py (int)". Replace "(int)" by any integer value. The variable will denote the number of epochs for which the model will be trained. The models will be saved in the Output folder in this directory. The weights and model after training for 70 epochs can be found here.

  4. After training execute "python test.py image" for generating a caption of an image. Pass the extension of the image along with the name of the image file for example, "python test.py beach.jpg". The image file must be present in the test folder.

NOTE - You can skip the training part by directly downloading the weights and model file and placing them in the Output folder since the training part wil take a lot of time if working on a non-GPU system. A GTX 1050 Ti with 4 gigs of RAM takes around 10-15 minutes for one epoch.

Output

The output of the model is a caption to the image and a python library called pyttsx which converts the generated text to audio

Results

Following are a few results obtained after training the model for 70 epochs.

Image Caption
Generated Caption: A brown dog is running in the water.
Generated Caption: A tennis player hitting the ball.
Generated Caption: A child in a helmet is riding a bike.
Generated Caption: A group of people are walking on a busy street.

On providing an ambiguous image for example a hamsters face morphed on a lion the model got confused but since the data is a bit biased towards dogs hence it captions it as a dog and the reddish pink nose of the hamster is identified as red ball

Image Caption
Generated Caption: A black dog is running through the snow with a red ball.

In some cases the classifier got confused and on blurring an image it produced bizzare results

Image Caption
Generated Caption: A brown dog and a brown dog are playing with a ball in the snow.
Generated Caption: A little girl in a white shirt is running on the grass.

References

NIC Model

[1] Vinyals, Oriol, et al. "Show and tell: A neural image caption generator." Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.

Data

https://illinois.edu/fb/sec/1713398

VGG16 Model

https://github.com/fchollet/deep-learning-models

Saved Model

https://drive.google.com/drive/folders/1aukgi_3xtuRkcQGoyAaya5pP4aoDzl7r

Code reference

https://github.com/anuragmishracse/caption_generator

You can find a detailed report in the Report folder.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].