AMSS-Net: Audio Manipulation on User-Specified Sources with Textual Queries

An official implementation of the paper: "AMSS-Net: Audio Manipulation on User-Specified Sources with Textual Queries"

• ArXiv link
• Demonstration link

This repository does not contain the complete source code yet.

We will upload codes sooner or later, after refactorization, for better readability.

News

Our paper has been accepted to ACMMM 2021 !

1. Installation

(Optional)

conda create -n amss
conda activate amss

(Install)

conda install pytorch=1.7.1 cudatoolkit=11.0 -c pytorch
conda install -c conda-forge ffmpeg librosa
conda install -c anaconda jupyter
pip install torchtext musdb museval pytorch_lightning wandb pydub pysndfx

Also, you have to install sox,

• for linux: conda install -c conda-forge sox
• for Windows: download

2. Dataset: Musdb18

1. Download

1. Full dataset

   • The entire dataset is hosted on Zenodo and requires that users request access.
   • The tracks can only be used for academic purposes.
   • They manually check requests.

• After your request is accepted, then you can download the full dataset!

2. or Sample Dataset

   • download sample version of MUSDB18 which includes 7s excerpts using this script

     import musdb
     musdb.DB(root='etc/musdb18_dev', download=True)

2. Generate wave files

• run this!

  musdbconvert <your_DIR> <target_DIR> 

• musdbconvert is automatically installed if you have installed musdb with:

  pip install musdb

3. Train script example

• AMSS-Net

python train.py --musdb_root ../../repos/musdb18_wav --pre_trained_word_embedding glove.6B.100d --embedding_dim 100 --task task2 --model isolasion_smpocm --n_fft 4096 --gpus 4 --distributed_backend ddp --sync_batchnorm True --save_top_k 3 --min_epochs 100 --num_head 6 --num_latent_source 8 --optimizer adam --batch_size 4 --enable_pl_optimizer True --train_loss spec_mse --val_loss raw_l1 --check_val_every_n_epoch 10 --lr 0.0001 --precision 16 --num_worker 32 --pin_memory True --seed 2020 --deterministic True --n_blocks 9 --run_id your_run_id --log wandb

3. Evaluation script example

auto_task2_eval.py --musdb_root ../../repos/musdb18_wav --ckpt_root etc/checkpoints/ --model isolasion_smpocm --cuda True --batch_size 8 --logger wandb