Tensorflow 2D and 3D pose estimation
Main Repository used:https://github.com/ildoonet/tf-pose-estimation
Repository used for 3D plotting:https://github.com/pyqtgraph/pyqtgraph
Dependencies
python3
tensorflow 1.4.1+
opencv3, protobuf, python3-tk
Requiremnets
argparse
matplotlib
scipy
tqdm
requests
fire
dill
git+https://github.com/ppwwyyxx/tensorpack.git
PyQt 4.7+, PySide, or PyQt5
NumPy
For 3D graphics: pyopengl and qt-opengl
Now from the main repository we get the following 3D points in this format
17 points of x-coordinates,y-coordinates,z-coordinates in 3 lists...(matplotlib version)
[[[ -5.04015825 -144.44469312 -181.38417344 -80.66403119 136.93109848
116.99966118 -11.16919163 2.62527356 -1.05079033 -4.14445959
-2.09431128 191.76896804 267.20449991 262.0793236 -193.72193764
-250.90957393 -265.76258367]
[ 86.11631527 95.79752119 -356.94342453 -513.95376014 76.43512815
-375.77265781 -518.52439623 194.44081889 188.62745907 89.32833065
152.24195646 178.49202036 175.542326 50.77858199 197.77633214
205.08800837 79.63985917]
[-189.40400846 -226.05239806 -5.45200828 -287.15627626 -254.17645491
-5.88772637 -285.68280011 26.52276139 263.97948822 362.16815799
461.79813566 243.24526949 -33.83331904 -248.24538611 228.03920304
-35.04000835 -248.7765299 ]]]
17 lists containing each point coordinates as a single list...(pyqt format for printing 3d pose)
[ -5.04015825 86.11631527 -189.40400846]
[-144.44469312 95.79752119 -226.05239806]
[-181.38417344 -356.94342453 -5.45200828]
[ -80.66403119 -513.95376014 -287.15627626]
[ 136.93109848 76.43512815 -254.17645491]
[ 116.99966118 -375.77265781 -5.88772637]
[ -11.16919163 -518.52439623 -285.68280011]
[ 2.62527356 194.44081889 26.52276139]
[ -1.05079033 188.62745907 263.97948822]
[ -4.14445959 89.32833065 362.16815799]
[ -2.09431128 152.24195646 461.79813566]
[191.76896804 178.49202036 243.24526949]
[267.20449991 175.542326 -33.83331904]
[ 262.0793236 50.77858199 -248.24538611]
[-193.72193764 197.77633214 228.03920304]
[-250.90957393 205.08800837 -35.04000835]
[-265.76258367 79.63985917 -248.7765299 ]
Using Pyqt library we plot these points and get the estimated 3D pose.
Results
We get all the 2D keypoints and those are connected with best possible straight lines. We also get the heat map, Vectormap-x and Vectormap-y Finally using these 2D keypoints we estimate the 3D pose These are some of the test images and their results.
Multiple Persons in a single image
Images with no humans - 3D pose can't be detected because there are not enough keypoints for pose estimation.Here is an examle of image of a cat
For more tested images please look into tested images directory in this folder.
Here is a demo of 2D pose Estimation using Webcam. https://www.youtube.com/watch?v=XNDlHkFQmIs&list=PLpORSxrB3kQwrQ1IaGNqyRX1RM9kwOxwo
Here is a demo of 3D pose Estimation using Webcam. https://youtu.be/_E2vrBo9z4o
Following Improvements can be done
Sometimes the 3D poses estimated by the model are not accurate but still it almost detecting every common pose of humans.For detecting more complicated images we can train the model on complex images and see whether it is detecting the pose or not.
This model is running on Nvidia Geforce 840M with 4GB memory and we are getting nearly 8 frames per second.For near real time dectection we can use higher graphics memory cards.
Camera used for testing the model is 5MP laptop webcam.We can use higher resolution camerad for filtering out the noise and get a clear image.