Sign Language Datasets
This repository includes TFDS data loaders for sign language datasets.
Installation
From Source
pip install git+https://github.com/sign-language-processing/datasets.gitPyPi
Not available. Need to add automatic publication on push.
Usage
We demonstrate a loading script for every dataset in examples/load.ipynb
Our config includes the option to choose the resolution and fps, for example:
import tensorflow_datasets as tfds
import sign_language_datasets.datasets
from sign_language_datasets.datasets.config import SignDatasetConfig
# Loading a dataset with default configuration
aslg_pc12 = tfds.load("aslg_pc12")
# Loading a dataset with custom configuration
config = SignDatasetConfig(name="videos_and_poses256x256:12",
version="3.0.0", # Specific version
include_video=True, # Download and load dataset videos
process_video=True, # Process videos to tensors, or only save path to video
fps=12, # Load videos at constant, 12 fps
resolution=(256, 256), # Convert videos to a constant resolution, 256x256
include_pose="holistic") # Download and load Holistic pose estimation
rwth_phoenix2014_t = tfds.load(name='rwth_phoenix2014_t', builder_kwargs=dict(config=config))Datasets
| Dataset | Videos | Poses | Versions |
|---|---|---|---|
| aslg_pc12 | N/A | N/A | 0.0.1 |
| rwth_phoenix2014_t | Yes | Holistic | 3.0.0 |
| autsl | Yes | OpenPose, Holistic | 1.0.0 |
| dgs_corpus | Yes | OpenPose, Holistic | 3.0.0 |
| how2sign | Yes | OpenPose | 1.0.0 |
| sign2mint | Yes | 1.0.0 | |
| signtyp | Links | 1.0.0 | |
| swojs_glossario | Yes | 1.0.0 | |
| SignBank | N/A | 1.0.0 | |
| wlasl | Failed | OpenPose | None |
| msasl | None | ||
| Video-Based CSL | None | ||
| RVL-SLLL ASL | None |
Data Interface
We follow the following interface wherever possible to make it easy to swap datasets.
{
"id": tfds.features.Text(),
"signer": tfds.features.Text() | tf.int32,
"video": tfds.features.Video(shape=(None, HEIGHT, WIDTH, 3)),
"depth_video": tfds.features.Video(shape=(None, HEIGHT, WIDTH, 1)),
"fps": tf.int32,
"pose": {
"data": tfds.features.Tensor(shape=(None, 1, POINTS, CHANNELS), dtype=tf.float32),
"conf": tfds.features.Tensor(shape=(None, 1, POINTS), dtype=tf.float32)
},
"gloss": tfds.features.Text(),
"text": tfds.features.Text()
}Why not Huggingface Datasets?
Huggingface datasets do not work well with videos. From the lack of native support of the video type, to lack of support of arbitrary tensors. Furthermore, they currently have memory leaks that prevent from saving even the smallest of video datasets.
Cite
@misc{moryossef2021datasets,
title={Sign Language Datasets},
author={Moryossef, Amit},
howpublished={\url{https://github.com/sign-language-processing/datasets}},
year={2021}
}