All Projects → Daniil-Osokin → Lightweight Human Pose Estimation.pytorch

Daniil-Osokin / Lightweight Human Pose Estimation.pytorch

Licence: apache-2.0
Fast and accurate human pose estimation in PyTorch. Contains implementation of "Real-time 2D Multi-Person Pose Estimation on CPU: Lightweight OpenPose" paper.

Programming Languages

python
139335 projects - #7 most used programming language

Projects that are alternatives of or similar to Lightweight Human Pose Estimation.pytorch

Gccpm Look Into Person Cvpr19.pytorch
Fast and accurate single-person pose estimation, ranked 10th at CVPR'19 LIP challenge. Contains implementation of "Global Context for Convolutional Pose Machines" paper.
Stars: ✭ 137 (-85.7%)
Mutual labels:  lightweight, human-pose-estimation, real-time
Lightweight Human Pose Estimation 3d Demo.pytorch
Real-time 3D multi-person pose estimation demo in PyTorch. OpenVINO backend can be used for fast inference on CPU.
Stars: ✭ 331 (-65.45%)
Mutual labels:  lightweight, human-pose-estimation, real-time
Pytorch realtime multi Person pose estimation
Pytorch version of Realtime Multi-Person Pose Estimation project
Stars: ✭ 205 (-78.6%)
Mutual labels:  human-pose-estimation, real-time
jeelizGlanceTracker
JavaScript/WebGL lib: detect if the user is looking at the screen or not from the webcam video feed. Lightweight and robust to all lighting conditions. Great for play/pause videos if the user is looking or not, or for person detection. Link to live demo.
Stars: ✭ 68 (-92.9%)
Mutual labels:  lightweight, real-time
Lightweight Segmentation
Lightweight models for real-time semantic segmentation(include mobilenetv1-v3, shufflenetv1-v2, igcv3, efficientnet).
Stars: ✭ 261 (-72.76%)
Mutual labels:  lightweight, real-time
FastPose
pytorch realtime multi person keypoint estimation
Stars: ✭ 36 (-96.24%)
Mutual labels:  real-time, human-pose-estimation
Deepstream pose estimation
This is a sample DeepStream application to demonstrate a human pose estimation pipeline.
Stars: ✭ 168 (-82.46%)
Mutual labels:  human-pose-estimation, real-time
Shelfnet Human Pose Estimation
Fast and accurate Human Pose Estimation using ShelfNet with PyTorch
Stars: ✭ 95 (-90.08%)
Mutual labels:  human-pose-estimation, real-time
Openpose train
Training repository for OpenPose
Stars: ✭ 381 (-60.23%)
Mutual labels:  human-pose-estimation, real-time
Mobilepose Pytorch
Light-weight Single Person Pose Estimator
Stars: ✭ 427 (-55.43%)
Mutual labels:  lightweight, real-time
Openpose unity plugin
OpenPose's Unity Plugin for Unity users
Stars: ✭ 446 (-53.44%)
Mutual labels:  human-pose-estimation, real-time
Trt pose
Real-time pose estimation accelerated with NVIDIA TensorRT
Stars: ✭ 525 (-45.2%)
Mutual labels:  human-pose-estimation, real-time
Openpose
OpenPose: Real-time multi-person keypoint detection library for body, face, hands, and foot estimation
Stars: ✭ 22,892 (+2289.56%)
Mutual labels:  human-pose-estimation, real-time
Keras realtime multi Person pose estimation
Keras version of Realtime Multi-Person Pose Estimation project
Stars: ✭ 728 (-24.01%)
Mutual labels:  human-pose-estimation, real-time
Lightcomm4j
Yet another lightweight asynchronous network library for java
Stars: ✭ 25 (-97.39%)
Mutual labels:  lightweight
Chatify
A Laravel package that allows you to add a complete user messaging system into your new/existing Laravel application.
Stars: ✭ 885 (-7.62%)
Mutual labels:  real-time
Larastrator
All in one - admin panel with tailwindcss and vuejs.
Stars: ✭ 25 (-97.39%)
Mutual labels:  lightweight
Pytorch Pose
A PyTorch toolkit for 2D Human Pose Estimation.
Stars: ✭ 932 (-2.71%)
Mutual labels:  human-pose-estimation
Tcodeedit
Lightweight and syntax hilighted UNICODE editor
Stars: ✭ 27 (-97.18%)
Mutual labels:  lightweight
Spimedb
EXPLORE & EDIT REALITY
Stars: ✭ 14 (-98.54%)
Mutual labels:  real-time

Real-time 2D Multi-Person Pose Estimation on CPU: Lightweight OpenPose

This repository contains training code for the paper Real-time 2D Multi-Person Pose Estimation on CPU: Lightweight OpenPose. This work heavily optimizes the OpenPose approach to reach real-time inference on CPU with negliable accuracy drop. It detects a skeleton (which consists of keypoints and connections between them) to identify human poses for every person inside the image. The pose may contain up to 18 keypoints: ears, eyes, nose, neck, shoulders, elbows, wrists, hips, knees, and ankles. On COCO 2017 Keypoint Detection validation set this code achives 40% AP for the single scale inference (no flip or any post-processing done). The result can be reproduced using this repository. This repo significantly overlaps with https://github.com/opencv/openvino_training_extensions, however contains just the necessary code for human pose estimation.

🔥 Check out our new work on accurate (and still fast) single-person pose estimation, which ranked 10th on CVPR'19 Look-Into-Person challenge.

🔥🔥 Check out our lightweight 3D pose estimation, which is based on Single-Shot Multi-Person 3D Pose Estimation From Monocular RGB paper and this work.

Table of Contents

Other Implementations

Requirements

  • Ubuntu 16.04
  • Python 3.6
  • PyTorch 0.4.1 (should also work with 1.0, but not tested)

Prerequisites

  1. Download COCO 2017 dataset: http://cocodataset.org/#download (train, val, annotations) and unpack it to <COCO_HOME> folder.
  2. Install requirements pip install -r requirements.txt

Training

Training consists of 3 steps (given AP values for full validation dataset):

  • Training from MobileNet weights. Expected AP after this step is ~38%.
  • Training from weights, obtained from previous step. Expected AP after this step is ~39%.
  • Training from weights, obtained from previous step and increased number of refinement stages to 3 in network. Expected AP after this step is ~40% (for the network with 1 refinement stage, two next are discarded).
  1. Download pre-trained MobileNet v1 weights mobilenet_sgd_68.848.pth.tar from: https://github.com/marvis/pytorch-mobilenet (sgd option). If this doesn't work, download from GoogleDrive.

  2. Convert train annotations in internal format. Run python scripts/prepare_train_labels.py --labels <COCO_HOME>/annotations/person_keypoints_train2017.json. It will produce prepared_train_annotation.pkl with converted in internal format annotations.

    [OPTIONAL] For fast validation it is recommended to make subset of validation dataset. Run python scripts/make_val_subset.py --labels <COCO_HOME>/annotations/person_keypoints_val2017.json. It will produce val_subset.json with annotations just for 250 random images (out of 5000).

  3. To train from MobileNet weights, run python train.py --train-images-folder <COCO_HOME>/train2017/ --prepared-train-labels prepared_train_annotation.pkl --val-labels val_subset.json --val-images-folder <COCO_HOME>/val2017/ --checkpoint-path <path_to>/mobilenet_sgd_68.848.pth.tar --from-mobilenet

  4. Next, to train from checkpoint from previous step, run python train.py --train-images-folder <COCO_HOME>/train2017/ --prepared-train-labels prepared_train_annotation.pkl --val-labels val_subset.json --val-images-folder <COCO_HOME>/val2017/ --checkpoint-path <path_to>/checkpoint_iter_420000.pth --weights-only

  5. Finally, to train from checkpoint from previous step and 3 refinement stages in network, run python train.py --train-images-folder <COCO_HOME>/train2017/ --prepared-train-labels prepared_train_annotation.pkl --val-labels val_subset.json --val-images-folder <COCO_HOME>/val2017/ --checkpoint-path <path_to>/checkpoint_iter_280000.pth --weights-only --num-refinement-stages 3. We took checkpoint after 370000 iterations as the final one.

We did not perform the best checkpoint selection at any step, so similar result may be achieved after less number of iterations.

Known issue

We observe this error with maximum number of open files (ulimit -n) equals to 1024:

  File "train.py", line 164, in <module>
    args.log_after, args.val_labels, args.val_images_folder, args.val_output_name, args.checkpoint_after, args.val_after)
  File "train.py", line 77, in train
    for _, batch_data in enumerate(train_loader):
  File "/<path>/python3.6/site-packages/torch/utils/data/dataloader.py", line 330, in __next__
    idx, batch = self._get_batch()
  File "/<path>/python3.6/site-packages/torch/utils/data/dataloader.py", line 309, in _get_batch
    return self.data_queue.get()
  File "/<path>/python3.6/multiprocessing/queues.py", line 337, in get
    return _ForkingPickler.loads(res)
  File "/<path>/python3.6/site-packages/torch/multiprocessing/reductions.py", line 151, in rebuild_storage_fd
    fd = df.detach()
  File "/<path>/python3.6/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/<path>/python3.6/multiprocessing/reduction.py", line 182, in recv_handle
    return recvfds(s, 1)[0]
  File "/<path>/python3.6/multiprocessing/reduction.py", line 161, in recvfds
    len(ancdata))
RuntimeError: received 0 items of ancdata

To get rid of it, increase the limit to bigger number, e.g. 65536, run in the terminal: ulimit -n 65536

Validation

  1. Run python val.py --labels <COCO_HOME>/annotations/person_keypoints_val2017.json --images-folder <COCO_HOME>/val2017 --checkpoint-path <CHECKPOINT>

Pre-trained model

The model expects normalized image (mean=[128, 128, 128], scale=[1/256, 1/256, 1/256]) in planar BGR format. Pre-trained on COCO model is available at: https://download.01.org/opencv/openvino_training_extensions/models/human_pose_estimation/checkpoint_iter_370000.pth, it has 40% of AP on COCO validation set (38.6% of AP on the val subset).

Conversion to OpenVINO format

  1. Convert PyTorch model to ONNX format: run script in terminal python scripts/convert_to_onnx.py --checkpoint-path <CHECKPOINT>. It produces human-pose-estimation.onnx.
  2. Convert ONNX model to OpenVINO format with Model Optimizer: run in terminal python <OpenVINO_INSTALL_DIR>/deployment_tools/model_optimizer/mo.py --input_model human-pose-estimation.onnx --input data --mean_values data[128.0,128.0,128.0] --scale_values data[256] --output stage_1_output_0_pafs,stage_1_output_1_heatmaps. This produces model human-pose-estimation.xml and weights human-pose-estimation.bin in single-precision floating-point format (FP32).

C++ Demo

To run the demo download Intel® OpenVINO™ Toolkit https://software.intel.com/en-us/openvino-toolkit/choose-download, install it and build the samples (Inferring Your Model with the Inference Engine Samples part). Then run <SAMPLES_BIN_FOLDER>/human_pose_estimation_demo -m <path_to>/human-pose-estimation.xml -i <path_to_video_file> for the inference on CPU.

Python Demo

We provide python demo just for the quick results preview. Please, consider c++ demo for the best performance. To run the python demo from a webcam:

  • python demo.py --checkpoint-path <path_to>/checkpoint_iter_370000.pth --video 0

Citation:

If this helps your research, please cite the paper:

@inproceedings{osokin2018lightweight_openpose,
    author={Osokin, Daniil},
    title={Real-time 2D Multi-Person Pose Estimation on CPU: Lightweight OpenPose},
    booktitle = {arXiv preprint arXiv:1811.12004},
    year = {2018}
}
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].