Motion Capture Using Joint Skeleton Tracking and Surface Estimation

Juergen Gall, Carsten Stoll, Edilson de Aguiar, Christian Theobalt, Bodo Rosenhahn, and Hans-Peter Seidel

Abstract

This paper proposes a method for capturing the performance of a human or an animal from a multi-view video sequence. Given an articulated template model and silhouettes from a multi-view image sequence, our approach recovers not only the movement of the skeleton, but also the possibly non-rigid temporal deformation of the 3D surface. While large scale deformations or fast movements are captured by the skeleton pose and approximate surface skinning, true small scale deformations or non-rigid garment motion are captured by fitting the surface to the silhouette. We further propose a novel optimization scheme for skeleton-based pose estimation that exploits the skeleton’s tree structure to split the optimization problem into a local one and a lower dimensional global one. We show on various sequences that our approach can capture the 3D motion of animals and humans accurately even in the case of rapid movements and wide apparel like skirts.

Images

a) Estimated skeleton. b) Skeleton-based deformation. c) Non-rigid surface deformation. d) Estimated skeleton and surface.

Motion Capture Using Joint Skeleton Tracking and Surface Estimation

Video (only results) ~30MB (AVI)

Reviewed CVPR 2009 Video ~95MB (AVI)

Video

Data

All data is only for research purposes. When using this data, please acknowledge the effort that went into data collection by referencing the corresponding paper. If you have questions concerning the data, please contact Juergen Gall. A simple demo program reads the meshes, calibration data, and the images. It projects the meshes onto the image plane as shown in the middle column of the examples above.

Handstand

Smoothed Meshes mesh_s.tar.gz

Meshes mesh.tar.gz

Calibration data.tar.gz

Silhouettes silh.tar.gz (9.9MB)

Images im_p1.tar.gz (1.7GB), im_p2.tar.gz (1.7GB)

Wheel

Smoothed Meshes mesh_s.tar.gz

Meshes mesh.tar.gz

Calibration data.tar.gz

Silhouettes silh.tar.gz (6.8MB)

Images im_p1.tar.gz (1.2GB), im_p2.tar.gz (1.2GB)

Dance

Smoothed Meshes mesh_s.tar.gz

Meshes mesh.tar.gz

Calibration data.tar.gz

Silhouettes silh.tar.gz (14MB)

Images im_p1.tar.gz (1.4GB), im_p2.tar.gz (1.4GB) im_p3.tar.gz (2.0GB)

Skirt

Smoothed Meshes mesh_s.tar.gz

Meshes mesh.tar.gz

Calibration data.tar.gz

Silhouettes silh.tar.gz (15MB)

Images im_p1.tar.gz (1.5GB), im_p2.tar.gz (1.5GB), im_p3.tar.gz (1.5GB), im_p4.tar.gz (1.5GB)

Dog

Smoothed Meshes mesh_s.tar.gz

Meshes mesh.tar.gz

Calibration data.tar.gz

Silhouettes silh.tar.gz (1MB)

Images im.tar.gz (0.5GB)

Publications

Gall J., Stoll C., de Aguiar E., Theobalt C., Rosenhahn B., and Seidel H.-P., Motion Capture Using Joint Skeleton Tracking and Surface Estimation (PDF), IEEE Conference on Computer Vision and Pattern Recognition (CVPR'09), 2009. © IEEE