GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → FrederikSchorr → Sign Language

FrederikSchorr / Sign Language

Licence: mit
Sign Language Recognition for Deaf People

Programming Languages

python
139335 projects - #7 most used programming language

Labels

deep-learningopencvconvolutional-neural-networkslstmkeras-tensorflow

Projects that are alternatives of or similar to Sign Language

Intelegent lock
lock mechanism with face recognition and liveness detection
Stars: ✭ 134 (+106.15%)
Mutual labels:  opencv, convolutional-neural-networks, keras-tensorflow
Predictive Maintenance Using Lstm
Example of Multiple Multivariate Time Series Prediction with LSTM Recurrent Neural Networks in Python with Keras.
Stars: ✭ 352 (+441.54%)
Mutual labels:  lstm, keras-tensorflow
Thesemicolon
This repository contains Ipython notebooks and datasets for the data analytics youtube tutorials on The Semicolon.
Stars: ✭ 345 (+430.77%)
Mutual labels:  convolutional-neural-networks, lstm
Pneumonia Detection From Chest X Ray Images With Deep Learning
Detecting Pneumonia in Chest X-ray Images using Convolutional Neural Network and Pretrained Models
Stars: ✭ 64 (-1.54%)
Mutual labels:  convolutional-neural-networks, keras-tensorflow
Bcdu Net
BCDU-Net : Medical Image Segmentation
Stars: ✭ 314 (+383.08%)
Mutual labels:  convolutional-neural-networks, lstm
Keras Anomaly Detection
Anomaly detection implemented in Keras
Stars: ✭ 335 (+415.38%)
Mutual labels:  convolutional-neural-networks, lstm
Cnn lstm ctc ocr
Tensorflow-based CNN+LSTM trained with CTC-loss for OCR
Stars: ✭ 464 (+613.85%)
Mutual labels:  convolutional-neural-networks, lstm
Deeppicar
Deep Learning Autonomous Car based on Raspberry Pi, SunFounder PiCar-V Kit, TensorFlow, and Google's EdgeTPU Co-Processor
Stars: ✭ 242 (+272.31%)
Mutual labels:  opencv, convolutional-neural-networks
Head Pose Estimation
Head pose estimation by TensorFlow and OpenCV
Stars: ✭ 698 (+973.85%)
Mutual labels:  opencv, convolutional-neural-networks
Deep Music Genre Classification
🎵 Using Deep Learning to Categorize Music as Time Progresses Through Spectrogram Analysis
Stars: ✭ 23 (-64.62%)
Mutual labels:  lstm, keras-tensorflow
Deepseqslam
The Official Deep Learning Framework for Route-based Place Recognition
Stars: ✭ 49 (-24.62%)
Mutual labels:  convolutional-neural-networks, lstm
Deeplearning.ai Assignments
Stars: ✭ 268 (+312.31%)
Mutual labels:  convolutional-neural-networks, lstm
Chinese-Word-Segmentation-in-NLP
State of the art Chinese Word Segmentation with Bi-LSTMs
Stars: ✭ 23 (-64.62%)
Mutual labels:  lstm, keras-tensorflow
Personality Detection
Implementation of a hierarchical CNN based model to detect Big Five personality traits
Stars: ✭ 338 (+420%)
Mutual labels:  convolutional-neural-networks, lstm
Rus-SpeechRecognition-LSTM-CTC-VoxForge
Распознавание речи русского языка используя Tensorflow, обучаясь на базе Voxforge
Stars: ✭ 50 (-23.08%)
Mutual labels:  lstm, keras-tensorflow
Easy Deep Learning With Keras
Keras tutorial for beginners (using TF backend)
Stars: ✭ 367 (+464.62%)
Mutual labels:  convolutional-neural-networks, lstm
Image Captioning
Image Captioning: Implementing the Neural Image Caption Generator with python
Stars: ✭ 52 (-20%)
Mutual labels:  convolutional-neural-networks, lstm
Sign Language Interpreter Using Deep Learning
A sign language interpreter using live video feed from the camera.
Stars: ✭ 157 (+141.54%)
Mutual labels:  opencv, convolutional-neural-networks
Person Detection And Tracking
A tensorflow implementation with SSD model for person detection and Kalman Filtering combined for tracking
Stars: ✭ 193 (+196.92%)
Mutual labels:  opencv, convolutional-neural-networks
Music recommender
Music recommender using deep learning with Keras and TensorFlow
Stars: ✭ 528 (+712.31%)
Mutual labels:  convolutional-neural-networks, keras-tensorflow
View All Similar Projects ➔

Sign Language Recognition

  • This prototype "understands" sign language for deaf people
  • Includes all code to prepare data (eg from ChaLearn dataset), extract features, train neural network, and predict signs during live demo
  • Based on deep learning techniques, in particular convolutional neural networks (including state-of-the-art 3D model) and recurrent neural networks (LSTM)
  • Built with Python, Keras+Tensorflow and OpenCV (for video capturing and manipulation)

For 10-slide presentation + 1-min demo video see here.

Requirements

This code requires at least

  • python 3.6.5
  • tensorflow 1.8.0
  • keras 2.2.0
  • opencv-python 3.4.1.15

For the training of the neural networks a GPU is necessary (eg aws p2.xlarge). The live demo works on an ordinary laptop (without GPU), eg MacBook Pro, i5, 8GB.

Get the video data

See here for overview of suitable data-sets for sign-language for deaf people: https://docs.google.com/presentation/d/1KSgJM4jUusDoBsyTuJzTsLIoxWyv6fbBzojI38xYXsc/edit#slide=id.g3d447e7409_0_0

Download the ChaLearn Isolated Gesture Recognition dataset here: http://chalearnlap.cvc.uab.es/dataset/21/description/ (you need to register first)

The ChaLearn video descriptions and labels (for train, validation and test data) can be found here: data_set/chalearn

prepare_chalearn.py is used to unzip the videos and sort them by labels (using Keras best-practise 1 folder = 1 label): folderstructure

Prepare the video data

Extract image frames from videos

frame.py extracts image frames from each video (using OpenCV) and stores them on disc.

See pipeline_i3d.py for the parameters used for the ChaLearn dataset:

  • 40 frames per training/test videos (on average 5 seconds duration = approx 8 frames per second)
  • Frames are resized/cropped to 240x320 pixels

Calculate optical flow

opticalflow.py calculates optical flow from the image frames of a video (and stores them on disc). See pipeline_i3d.py for usage.

Optical flow is very effective for this type of video classification, but also very calculation intensive, see here.

Train the neural network

train_i3d.py trains the neural network. First only the (randomized) top layers are trained, then the entire (pre-trained) network is fine-tuned.

A pre-trained 3D convolutional neural network, I3D, developed in 2017 by Deepmind is used, see here and model_i3d.py.

Training requires a GPU and is performed through a generator which is provided in datagenerator.py.

Note: the code files containing "_mobile_lstm" are used for an alternative NN architecture, see here.

Predict during live demo

livedemo.py launches the webcam,

  • waits for the start signal from user,
  • captures 5 seconds of video (using videocapture.py),
  • extracts frames from the video
  • calculates and displays the optical flow,
  • and uses the neural network to predict the sign language gesture.

The neural network model is not included in this GitHub repo (too large) but can be downloaded here (150 MB).

License

This project is licensed under the MIT License - see the LICENSE file for details

Acknowledgments

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].