Awesome Speech Enhancement

Table of contents

• Overview
• Publications
  • Survey
  • Feature augmentation
  • Audio-Visual SE
  • Network design
    • Filter design
    • Fusion techniques
    • Attention
    • U-net
    • GAN
    • Auto-Encoder
  • Phase reconstruction
  • Learning strategy
    • Loss function
    • Multi-task learning
    • Curriculum learning
    • Transfer learning
  • Other improvements
• Datasets
• Tools
  • Framework
  • Evaluation
  • Audio feature extraction
  • Audio data augmentation
• SOTA results
• Applications
• Learning materials
  • Book or thesis
  • Video
  • Slides

Overview

This is a curated list of awesome Speech Enhancement tutorials, papers, libraries, datasets, tools, scripts and results. The purpose of this repo is to organize the world’s resources for speech enhancement, and make them universally accessible and useful.

This repo is jointly contributed by Nana Hou (Nanyang Technoligical University), Meng Ge, Hao Shi (Tianjin University), Chenglin Xu (National University of Singapore), Chen Weiguang (Hunan University).

To add items to this page, simply send a pull request.

Publications

Coming soon...

Survey

• A literature survey on single channel speech enhancement, 2020 [paper]
• Research Advances and Perspectives on the Cocktail Party Problem and Related Auditory Models, Bo Xu, 2019 [paper (Chinese)]
• Deep Learning for Environmentally Robust Speech Recognition: An Overview of Recent Developments, Zixing Zhang, 2017 [paper]
• Supervised speech separation based on deep learning: An Overview, 2017 [paper]
• A review on speech enhancement techniques, 2015 [paper]
• Nonlinear speech enhancement: an overview, 2007 [paper]

Feature augmentation

• Speech enhancement using self-adaptation and multi-head attention, ICASSP 2020 [paper]
• PAN: phoneme-aware network for monaural speech enhancement, ICASSP 2020 [paper]
• Noise tokens: learning neural noise templates for environment-aware speech enhancement [paper]
• Speaker-aware deep denoising autoencoder with embedded speaker identity for speech enhancement, Interspeech 2019 [paper]

Audio-Visual SE

• Lite Audio-Visual Speech Enhancement, INTERSPEECH 2020 [paper]
• Audio-Visual Speech Enhancement Using Multimodal Deep Convolutional Neural Networks, TETCI, 2018 (first audio-visual SE) [journal]

Network design

Filter design

• Efficient trainable front-ends for neural speech enhancement, ICASSP 2020 [paper]

Fusion techniques

• Spectrograms fusion with minimum difference masks estimation for monaural speech dereverberation, ICASSP 2020 [paper]
• Masking and inpainting: a two-stage speech enhancement approach for low snr and non-stationary noise, ICASSP 2020 [paper]
• A composite dnn architecture for speech enhancement, ICASSP 2020 [paper]
• An attention-based neural network approach for single channel speech enhancement, ICASSP 2019 [paper]
• Multi-domain processing via hybrid denoising networks for speech enhancement, 2018 [paper]

Attention

• Speech enhancement using self-adaptation and multi-head attention, ICASSP 2020 [paper]
• Channel-attention dense u-net for multichannel speech enhancement, ICASSP 2020 [paper]
• T-GSA: transformer with gaussian-weighted self-attention for speech enhancement, ICASSP 2020 [paper]

U-net

• Phase-aware speech enhancement with deep complex u-net, ICLR 2019 [paper] [code]

GAN

• PAGAN: a phase-adapted generative adversarial networks for speech enhancement, ICASSP 2020 [paper]
• MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancement, ICML 2019 [paper]
• Time-frequency masking-based speech enhancement using generative adversarial network, ICASSP 2018 [paper]
• SEGAN: speech enhancement generative adversarial network, Interspeech 2017 [paper]

Auto-Encoder

• Speech Enhancement Based on Deep Denoising Autoencoder, INTERSPEECH 2013 (first deep learning based SE) [paper]

Phase reconstruction

• Phase reconstruction based on recurrent phase unwrapping with deep neural networks, ICASSP 2020 [paper]
• PAGAN: a phase-adapted generative adversarial networks for speech enhancement, ICASSP 2020 [paper
• Invertible dnn-based nonlinear time-frequency transform for speech enhancement, ICASSP 2020 [paper]
• Phase-aware speech enhancement with deep complex u-net, ICLR 2019 [paper] [code]
• PHASEN: A Phase-and-Harmonics-Aware Speech Enhancement Network, AAAI 2020 [paper]

Learning strategy

Loss function

• MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancement, ICML 2019 [paper]
• Speech denoising with deep feature losses, Interspeech 2019 [paper]
• End-to-end multi-task denoising for joint sdr and pesq optimization, Arxiv 2019 [paper]

Multi-task learning

• Multi-objective learning and mask-based post-processing for deep neural network based speech enhancement, Arxiv 2017 [paper]
• Multiple-target deep learning for LSTM-RNN based speech enhancement, HSCMA 2017 [paper]
• Speech enhancement and recognition using multi-task learning of long short-term memory recurrent neural networks, ISCA 2015 [paper]

Curriculum learning

• SNR-Based Progressive Learning of Deep Neural Network for Speech Enhancement, INTERSPEECH 2016 [paper]

Transfer learning

• Cross-language transfer learning for deep neural network based speech enhancement, ISCSLP 2014 [paper]

Other improvements

• Improving robustness of deep learning based monaural speech enhancement against processing artifacts, ICASSP 2020 [paper]

Tools

Framework

Link	Language	Description
SETK	Python & C++	SETK: Speech Enhancement Tools integrated with Kaldi.
pyAudioAnalysis	Python	Python Audio Analysis Library: Feature Extraction, Classification, Segmentation and Applications.
Beamformer	Python	Implementation of the mask-based adaptive beamformer (MVDR, GEVD, MCWF).
Time-frequency Mask	Python	Computation of the time-frequency mask (PSM, IRM, IBM, IAM, ...) as the neural network training labels.
SSL	Python	Implementation of Sound Source Localization.
Data format	Python	Format tranform between Kaldi, Numpy and Matlab.

Evaluation

Audio feature extraction

Link	Language	Description
LPS	Python	Extract log-power-spectrum/magnitude spectrum/log-magnitude spectrum/Cepstral mean and variance normalization.
MFCC	Python	This library provides common speech features for ASR including MFCCs and filterbank energies.
pyAudioAnalysis	Python	Python Audio Analysis Library: Feature Extraction, Classification, Segmentation and Applications.

Audio data augmentation

Link	Language	Description
Data simulation	Python	Add reverberation, noise or mix speaker.
RIR simulation	Python	Generation of the room impluse response (RIR) using image method.
pyroomacoustics	Python	Pyroomacoustics is a package for audio signal processing for indoor applications.
gpuRIR	Python	Python library for Room Impulse Response (RIR) simulation with GPU acceleration
rir_simulator_python	Python	Room impulse response simulator using python
audiomentations	Python	A Python library for audio data augmentation, e.g. time stretch, pitch shift, add noise, add room reverberation

Datasets

Speech ehancement datasets (sorted by usage frequency in paper)

Augmentation noise sources (sorted by usage frequency in paper)

SOTA results

STOA results in dataset by University of Edinburgh. The following methods are all trained by "trainset_28spk" and tested by common testset. ("F" denotes frequency-domain and "T" is time-domain.)

Applications

• a speech enhancement Android APP from Prof. Yu Tsao's group [download][video][Github]

Learning materials

Book or thesis

• Audio Source Separation and Speech Enhancement, Emmanuel Vincent, 2019 [link]
• A Study on WaveNet, GANs and General CNNRNN Architectures, 2019 [link]
• Deep learning: method and applications, 2016 [link]
• Deep learning by Ian Goodfellow and Yoshua Bengio and Aaron Courville, 2016 [link]
• Robust automatic speech recognition by Jinyu Li and Li Deng, 2015 [link]

Video

• CCF speech seminar 2020 [link]
• Real-time Single-channel Speech Enhancement with Recurrent Neural Networks by Microsoft Research, 2019 [link]
• Deep learning in speech by Hongyi Li, 2019 [link]
• High-Accuracy Neural-Network Models for Speech Enhancement, 2017 [link]
• DNN-Based Online Speech Enhancement Using Multitask Learning and Suppression Rule Estimation, 2015 [link]
• Microphone array signal processing: beyond the beamformer,2011 [link]

Slides

• Intelligibility Evaluation and Speech Enhancement based on Deep Learning by Yu Tsao, (INTERSPEECH 2020 tutorial) [link] [video]
• Speech Enhancement based on Deep Learning and Intelligibility Evaluation by Yu Tsao, (APSIPA 2019 tutorial) [link]
• Deep learning in speech by Hongyi Li, 2019 [link]
• Learning-based approach to speech enhancement and separation (INTERSPEECH tutorial, 2016) [link]
• Deep learning for speech/language processing (INTERSPEECH tutorial by Li Deng, 2015) [link]
• Speech enhancement algorithms (Stanford University, 2013) [link]