All Projects β†’ xrenaa β†’ Music Dance Video Synthesis

xrenaa / Music Dance Video Synthesis

(ACM MM 20 Oral) PyTorch implementation of Self-supervised Dance Video Synthesis Conditioned on Music

Programming Languages

python
139335 projects - #7 most used programming language

Projects that are alternatives of or similar to Music Dance Video Synthesis

Awesome Face
😎 face releated algorithm, dataset and paper
Stars: ✭ 739 (+392.67%)
Mutual labels:  dataset, paper
Vehicle reid Collection
πŸš— the collection of vehicle re-ID papers, datasets. πŸš—
Stars: ✭ 225 (+50%)
Mutual labels:  dataset, paper
Body reconstruction references
Paper, dataset and code collection on human body reconstruction
Stars: ✭ 96 (-36%)
Mutual labels:  dataset, paper
Surface Defect Detection
πŸŽπŸ“ˆ Constantly summarizing open source dataset and important critical papers in the field of surface defect research which are very important. πŸ‹
Stars: ✭ 287 (+91.33%)
Mutual labels:  dataset, paper
Recursive Cnns
Implementation of my paper "Real-time Document Localization in Natural Images by Recursive Application of a CNN."
Stars: ✭ 80 (-46.67%)
Mutual labels:  dataset, paper
Core50
CORe50: a new Dataset and Benchmark for Continual Learning
Stars: ✭ 91 (-39.33%)
Mutual labels:  dataset, paper
Dataset
Crop/Weed Field Image Dataset
Stars: ✭ 98 (-34.67%)
Mutual labels:  dataset, paper
Netcdf Fortran
Official GitHub repository for netCDF-Fortran libraries, which depend on the netCDF C library. Install the netCDF C library first.
Stars: ✭ 141 (-6%)
Mutual labels:  dataset
Vditor
♏ δΈ€ζ¬Ύζ΅θ§ˆε™¨η«―ηš„ Markdown 编辑器。
Stars: ✭ 1,742 (+1061.33%)
Mutual labels:  multimedia
E2e Model Learning
Task-based end-to-end model learning in stochastic optimization
Stars: ✭ 140 (-6.67%)
Mutual labels:  paper
Prosody
Helsinki Prosody Corpus and A System for Predicting Prosodic Prominence from Text
Stars: ✭ 139 (-7.33%)
Mutual labels:  dataset
Triggerner
TriggerNER: Learning with Entity Triggers as Explanations for Named Entity Recognition (ACL 2020)
Stars: ✭ 141 (-6%)
Mutual labels:  dataset
Dstc7 End To End Conversation Modeling
Grounded conversational dataset for end-to-end conversational AI (official DSTC7 data)
Stars: ✭ 141 (-6%)
Mutual labels:  dataset
Coffee Quality Database
Building the Coffee Quality Institute Database
Stars: ✭ 141 (-6%)
Mutual labels:  dataset
Opentraj
Human Trajectory Prediction Dataset Benchmark (ACCV 2020)
Stars: ✭ 144 (-4%)
Mutual labels:  dataset
Ml Datasets
Machine Learning datasets for Nepal
Stars: ✭ 139 (-7.33%)
Mutual labels:  dataset
Ffmpeg Video Player
An FFmpeg and SDL Tutorial.
Stars: ✭ 149 (-0.67%)
Mutual labels:  multimedia
Face Detect
A Python based tool to extract faces from any picture.
Stars: ✭ 146 (-2.67%)
Mutual labels:  dataset
Baidutraffic
This repo includes introduction, code and dataset of our paper Deep Sequence Learning with Auxiliary Information for Traffic Prediction (KDD 2018).
Stars: ✭ 143 (-4.67%)
Mutual labels:  dataset
Paper
🌈 δΈ€δΈͺη±»ηΊΈι£Žηš„δΈ»ι’˜paperπŸŽ‰(still updating...)
Stars: ✭ 142 (-5.33%)
Mutual labels:  paper

Self-supervised Dance Video Synthesis Conditioned on Music

Pytorch implementation for this paper by Xuanchi Ren, Haoran Li, Zijian Huang, Qifeng Chen

To appear in ACM MM 2020

[Paper] [Paper_MM]

The demo video is shown at: https://youtu.be/UNHv7uOUExU

The dataset and the code for training and test is released.

A notebook for demo and quick start will be provided soon.

Some Demo:

More samples can be seen in demo video.

Requirement:

python 3.5 + pytorch 1.0

For the Testing part, you should install ffmpeg for music video.

We use tensorboardX for logging. If you don't install it, you can just comment the line in train.py.

Training:

This training process is intended for the clean part dataset, which could be downloaded here.

  1. Download the dataset and put it under ./dataset

  2. Run

python train.py

training script will load config of config.py. If you want to train the model on other datasets, you should change the config in config.py.

Testing:

If you want to use the pretrained model, you can firstly download it from here, put it under "pretrain_model" and change the path of get_demo.py to "./pretrain_model/generator_0400.pth".

  1. Run
python get_demo.py --output the_output_path
  1. Make the output skeleton sequence to music video
cd Demo
./frame2vid.sh

Note that you should change the paths and the "max" variable in frame2vid.sh.

Pose2Vid:

For this part, we adapt the method of the paper "Everybody dance now".

And We use this pytorch implementation.

Metrics:

For the proposed cross-modal metric in our paper, we re-implement the paper: Human Motion Analysis with Deep Metric Learning (ECCV 2018).

The implementation of this paper can be seen at: https://github.com/xrenaa/Human-Motion-Analysis-with-Deep-Metric-Learning

Dataset:

To use the dataset, please refer the notebook "dataset/usage_dataset.ipynb"

As state in the paper, we collect 60 videos in total, and divide them into 2 part according to the cleaness of the skeletons.

The clean part(40 videos): https://drive.google.com/file/d/1o79F2F7-dZ7Cvpzf6hsVMwvfNg9LM3_K/view?usp=sharing

The noisy part(20 videos): https://drive.google.com/file/d/1pZ3JszX7393dQwm6x6bxxbiKb0wLIJGE/view?usp=sharing

To support further study, we also provide other collected data:

Ballet: https://drive.google.com/open?id=1NR6S20EI1C37fsDhaNkRI_P1MLT9Ox7u

Popping: https://drive.google.com/file/d/1oLIxtczDZBvPdCAk8wuiI9b4FsnCItMG/view?usp=sharing

Boy_kpop: https://drive.google.com/file/d/14-kEdudvaGLSapAr4prp4D67wzyACVQt/view?usp=sharing

Besides, we also provide the BaiduNetDisk version: https://pan.baidu.com/s/15wLkdPnlZiCxgnPv51hgpg (includes all the dataset)

Questions

If you have questions for our work, please email to [email protected].

Citation

If you use this code for your research, please cite our paper.

@InProceedings{ren_mm_dance,
author = {Xuanchi Ren, Haoran Li, Zijian Huang, Qifeng Chen},
title = {Self-supervised Dance Video Synthesis Conditioned on Music},
booktitle = {ACM MM},
year = {2020}
}
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].