All Projects → drscotthawley → Panotti

drscotthawley / Panotti

Licence: mit
A multi-channel neural network audio classifier using Keras

Programming Languages

python
139335 projects - #7 most used programming language

Projects that are alternatives of or similar to Panotti

Im2latex Tensorflow
Tensorflow implementation of the HarvardNLP paper - What You Get Is What You See: A Visual Markup Decompiler (https://arxiv.org/pdf/1609.04938v1.pdf)
Stars: ✭ 207 (-15.16%)
Mutual labels:  convolutional-neural-networks
Xsum
Topic-Aware Convolutional Neural Networks for Extreme Summarization
Stars: ✭ 221 (-9.43%)
Mutual labels:  convolutional-neural-networks
Cnn Text Classification Tf Chinese
CNN for Chinese Text Classification in Tensorflow
Stars: ✭ 237 (-2.87%)
Mutual labels:  convolutional-neural-networks
Transfer Learning Suite
Transfer Learning Suite in Keras. Perform transfer learning using any built-in Keras image classification model easily!
Stars: ✭ 212 (-13.11%)
Mutual labels:  convolutional-neural-networks
Deep Alignment Network A Convolutional Neural Network For Robust Face Alignment
This is a Tensorflow implementations of paper "Deep Alignment Network: A convolutional neural network for robust face alignment".
Stars: ✭ 219 (-10.25%)
Mutual labels:  convolutional-neural-networks
Triplet Attention
Official PyTorch Implementation for "Rotate to Attend: Convolutional Triplet Attention Module." [WACV 2021]
Stars: ✭ 222 (-9.02%)
Mutual labels:  convolutional-neural-networks
Style transfer
CNN image style transfer 🎨.
Stars: ✭ 210 (-13.93%)
Mutual labels:  convolutional-neural-networks
Image Super Resolution
🔎 Super-scale your images and run experiments with Residual Dense and Adversarial Networks.
Stars: ✭ 3,293 (+1249.59%)
Mutual labels:  convolutional-neural-networks
Retrieval 2017 Cam
Class-Weighted Convolutional Features for Image Retrieval (BMVC 2017)
Stars: ✭ 219 (-10.25%)
Mutual labels:  convolutional-neural-networks
Super Slomo
PyTorch implementation of Super SloMo by Jiang et al.
Stars: ✭ 2,714 (+1012.3%)
Mutual labels:  convolutional-neural-networks
Tfwss
Weakly Supervised Segmentation with Tensorflow. Implements instance segmentation as described in Simple Does It: Weakly Supervised Instance and Semantic Segmentation, by Khoreva et al. (CVPR 2017).
Stars: ✭ 212 (-13.11%)
Mutual labels:  convolutional-neural-networks
3dmmasstn
MatConvNet implementation for incorporating a 3D Morphable Model (3DMM) into a Spatial Transformer Network (STN)
Stars: ✭ 218 (-10.66%)
Mutual labels:  convolutional-neural-networks
Pyconv
Pyramidal Convolution: Rethinking Convolutional Neural Networks for Visual Recognition (https://arxiv.org/pdf/2006.11538.pdf)
Stars: ✭ 231 (-5.33%)
Mutual labels:  convolutional-neural-networks
Dynamic Training Bench
Simplify the training and tuning of Tensorflow models
Stars: ✭ 210 (-13.93%)
Mutual labels:  convolutional-neural-networks
Mead Baseline
Deep-Learning Model Exploration and Development for NLP
Stars: ✭ 238 (-2.46%)
Mutual labels:  convolutional-neural-networks
Cnn From Scratch
A scratch implementation of Convolutional Neural Network in Python using only numpy and validated over CIFAR-10 & MNIST Dataset
Stars: ✭ 210 (-13.93%)
Mutual labels:  convolutional-neural-networks
Master Thesis Bayesiancnn
Master Thesis on Bayesian Convolutional Neural Network using Variational Inference
Stars: ✭ 222 (-9.02%)
Mutual labels:  convolutional-neural-networks
Torchfusion
A modern deep learning framework built to accelerate research and development of AI systems
Stars: ✭ 244 (+0%)
Mutual labels:  convolutional-neural-networks
Deeppicar
Deep Learning Autonomous Car based on Raspberry Pi, SunFounder PiCar-V Kit, TensorFlow, and Google's EdgeTPU Co-Processor
Stars: ✭ 242 (-0.82%)
Mutual labels:  convolutional-neural-networks
Tensorflow 101
TensorFlow Tutorials
Stars: ✭ 2,565 (+951.23%)
Mutual labels:  convolutional-neural-networks

DOI

Panotti: A Convolutional Neural Network classifier for multichannel audio waveforms

Panotti image (Image of large-eared Panotti people, Wikipedia)

This is a version of the audio-classifier-keras-cnn repo (which is a hack of @keunwoochoi's compact_cnn code). Difference with Panotti is, it has been generalized beyond mono audio, to include stereo or even more "channels." And it's undergone many refinements.

NOTE: The majority of issues people seem to have in using this utility, stem from inconsistencies in their audio datasets. This is to the point where I hesitate to delve into such reports. I suggest trying the binaural audio example and see if your same problems arise. -SH

Installation

UPDATE June 9, 2020: There is an updated version of Panotti that works with TensorFlow 2, currently in the panotti branch called 'tf2'. I'm not ready to merge that branch with master until Vibrary is also updated for TF2.

Preface: Requirements

Probably Mac OS X or Linux. (Windows users: I have no experience to offer you.) Not everything is required, here's a overview:

  • Required:
    • Python 3.5
    • numpy
    • keras
    • tensorflow
    • librosa
    • matplotlib
    • h5py
  • Optional:
    • sox ("Sound eXchange": command-line utility for examples/binaural. Install via "apt-get install sox")
    • pygame (for exampes/headgames.py)
    • For sorting-hat: flask, kivy kivy-garden

...the requirements.txt file method is going to try to install both required and optional packages.

Installation:

git clone https://github.com/drscotthawley/panotti.git

cd panotti

pip install -r requirements.txt

Demo

I'm not shipping this with any audio but you can generate some for the 'fake binaural' example (requires sox):

cd examples
./binaural_setup.sh
cd binaural
../../preprocess_data.py --dur=2 --clean
../../train_network.py

Quick Start

  • Make a folder called Samples/ and inside it create sub-folders with the names of each category you want to train on. Place your audio files in these sub-folders accordingly.
  • run python preprocess_data.py
  • run python train_network.py
  • run python eval_network.py - This applies the trained network to the testing dataset and gives you accuracy reports.

Data Preparation

Data organization:

Sound files should go into a directory called Samples/ that is local off wherever the scripts are being run. Within Samples, you should have subdirectories which divide up the various classes.

Example: for the IDMT-SMT-Audio-Effects database, using their monophonic guitar audio clips...

$ ls -F Samples/
Chorus/  Distortion/  EQ/  FeedbackDelay/  Flanger/   NoFX/  Overdrive/  Phaser/  Reverb/  SlapbackDelay/
Tremolo/  Vibrato/
$

(Within each subdirectory of Samples, there are loads of .wav or .mp3 files that correspond to each of those classes.)

"Is there any sample data that comes with this repo?" Not the data itself, but check out the examples/ directory. ;-)

Data augmentation & preprocessing:

(Optional) Augmentation:

The "augmentation" will vary the speed, pitch, dynamics, etc. of the sound files ("data") to try to "bootstrap" some extra data with which to train. If you want to augment, then you'll run it as

$ python augment_data.py <N> Samples/*/*

where N is how many augmented copies of each file you want it to create. It will place all of these in the Samples/ directory with some kind of "_augX" appended to the filename (where X just counts the number of the augmented data files). For augmentation it's assumed that all data files have the same length & sample rate.

(Required) Preprocessing:

When you preprocess, the data-loading will go much faster (e.g., 100 times faster) the next time you try to train the network. So, preprocess.

Preprocessing will pad the files with silence to fit the length to the length of the longest file and the number of channels to the file with the most channels. It will then generate mel-spectrograms of all data files, and create a "new version" of Samples/ called Preproc/.

It will do an 80-20 split of the dataset, so within Preproc/ will be the subdirectories Train/ and Test/. These will have the same subdirectory names as Samples/, but all the .wav and .mp3 files will have ".npy" on the end now. Datafiles will be randomly assigned to Train/ or Test/, and there they shall remain.

To do the preprocessing you just run

$ python preprocess_data.py

Training & Evaluating the Network

$ python train_network.py That's all you need. (I should add command-line arguments to adjust the layer size and number of layers...later.)

It will perform an 80-20 split of training vs. testing data, and give you some validation scores along the way.

It's set to run for 2000 epochs, feel free to shorten that or just ^C out at some point. It automatically does checkpointing by saving(/loading) the network weights via a new file weights.hdf5, so you can interrupt & resume the training if you need to.

After training, more diagnostics -- ROC curves, AUC -- can be obtained by running

$ python eval_network.py

(Changing the batch_size variable between training and evaluation may not be a good idea. It will probably screw up the Batch Normalization...but maybe you'll get luck.)

Results

On the IDMT Audio Effects Database using the 20,000 monophonic guitar samples across 12 effects classes, this code achieved 99.7% accuracy and an AUC of 0.9999. Specifically, 11 mistakes were made out of about 4000 testing examples; 6 of those were for the 'Phaser' effect, 3 were for EQ, a couple elsewhere, and most of the classes had zero mistakes. (No augmentation was used.)

This accuracy is comparable to the original 2010 study by Stein et al., who used a Support Vector Machine.

This was achieved by running for 10 hours on our workstation with an NVIDIA GTX1080 GPU.

Extra Tricks

  • We have multi-GPU training. The saving & loading means we get warning messages from Keras. Ignore those. It's because if we compile both the parallel model and its serial counterpart, it breaks things. So we leave the serial one uncompiled and that's the one we have to save. I regard this problem as a 'bug' in the Keras multi-gpu protocols.
  • Speaking of saving & loading, we encode the names of the output classes in the weights.hdf5 file using a HDF5 attribute 'class_names'.

-- [@drscotthawley](https://drscotthawley.github.io)
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].