Vision to Dynamics

Synchronized 3D Motion Capture and 2D Video Capture

PSU-TMM100 Data Modalities

Description: The PSU Taiji MultiModal (PSU-TMM100) Dataset contains 100 Taiji motion sequences, 24-form Simplified Taiji-Quan, aka Tai Chi, performed by 10 human subjects. The dataset includes time-synchronized measurements:

- motion capture markers and body joints recorded using Vicon Nexus 2.6.1

- foot pressure recorded using Tekscan F-scan 7.0 insole sensors

- 1080p video from two views recorded using Vicon Nexus 2.6.1

Additionally, OpenPose and HRNet pose detection networks networks are applied on both video views to generate 2D and triangulated 3D vision-based joints estimates.

If you use this dataset for publication, we kindly ask you to attribute credit by citing the ECCV2020 Paper providing detailed description of the dataset and benchmarks:

@inproceedings{Scott2020,
  title = {From Image to Stability: Learning Dynamics from Human Pose},
  author = {Scott, Jesse and Ravichandran, Bharadwaj and Funk, Christopher and Collins, Robert T. and Liu, Yanxi},
  booktitle = {Computer Vision -- ECCV 2020},
  pages = {536--554},
  isbn = {978-3-030-58592-1},
  year = {2020},
  editor = {Vedaldi, Andrea and Bischof, Horst and Brox, Thomas and Frahm, Jan-Michael},
  publisher = {Springer International Publishing},
  address = {Cham},
  doi = {10.1007/978-3-030-58592-1_32},
}

Access Requests: Dataset Request Form

Additional contacts: Keaton Kraiger, Yanxi Liu

This dataset collection is carried out under PSU IRB STUDY8085, by members of PSU LPAC, primarily by Jesse Scott advised by professors Robert Collins (CSE), John Challis (Kinesiology), and Yanxi Liu (EE and CSE). This research is supported in part by NSF grant IIS-1218729, NSF grant IIS-1909315, and the Penn State College of Engineering Dean's office.

Abstract

Pose stability analysis is the key to understanding locomotion and control of body equilibrium, with applications in numerous fields such as kinesiology, medicine, and robotics. In biomechanics, Center of Pressure (CoP) is used in studies of human postural control and gait. We propose a novel approach to learn CoP from pose of a human body to aid stability analysis, learning foot pressure heatmaps (dynamics) from 2D human pose (kinematics) derived from video. We validated our learned results on a collection of long (5min +) choreographed Taiji (Tai Chi) sequences of multiple subjects with synchronized foot pressure and video data. Cross-subject validation results show promising performance, significantly outperforming the baseline methods. Furthermore, we demonstrate that our computation of center of pressure (CoP) approaches the expectations of corresponding lab-based measurements.

Publications

(03/23) A Light-Weight Contrastive Approach for Aligning Human Pose Sequences - Robert T. Collins - arXiv Preprint - arXiv

(12/22) Image-Based Stability Quantification - Jesse Scott, John Challis, Robert T. Collins and Yanxi Liu - IEEE Transactions on Neural Systems and Rehabilitation Engineering - Online Preview

(08/22) Dynamic Stability Monitoring of Complex Human Motion Sequences via Precision Computer Vision - Jesse Scott - Ph.D. Dissertation - PDF

(06/22) Image-Based Stability Quantification - Jesse Scott, John Challis, Robert T. Collins and Yanxi Liu - IEEE Transactions on Neural Systems and Rehabilitation Engineering - arXiv

(08/20) From Image to Stability: Learning Dynamics from Human Pose - Jesse Scott*, Bharadwaj Ravichandran*, Christopher Funk, Robert T. Collins and Yanxi Liu - European Conference on Computer Vision (ECCV) - *Co-First Authors - PDF(ResearchGate)

ECCV 2020 Short Presentation Video

ECCV 2020 Full Presentation Video

Example Results

(04/20) BioPose-3D and PressNet-KL: A Path to Understanding Human Pose Stability from Video - Bharadwaj Ravichandran - M.S. Thesis - PDF

(01/20) From Kinematics To Dynamics: Estimating Center of Pressure and Base of Support from Video Frames of Human Motion Jesse Scott, Christopher Funk, Bharadwaj Ravichandran, John H. Challis, Robert T. Collins, and Yanxi Liu - arXiv

(08/18) 3D Human Pose Estimation on Taiji Sequence - Tianhe Wang - M.S. Thesis - PDF

(05/18) Multiple Object Tracking - Aiyu Cui - B.S. Honors Thesis - PDF

(10/17) 4D Model-based Spatiotemporal Alignment of Scripted Taiji Quan Sequences - PDF(ResearchGate)

Located in the Supplemental Mail Building on Penn State University, University Park Campus, the Motion Capture Lab is a College of Engineering Supported facility jointly utilized by:

- Laboratory for Perception, Action, and Cognition (LPAC),

- Air Vehicle Intelligence and Autonomy Laboratory (AVIA)and

- Vertical Lift Research Center of Excellence (VLRCOE).

The facility is a class B commercial space that facilitates a motion capture volume that is 25x20x15 foot with smooth concrete floor and concrete block walls on 3 sides with safety netting on the 4th wall to protect the system operator and research technician.

For motion capture and video data recording, the laboratory uses a Vicon motion capture system with 12 T40 motion capture cameras and 2 Vue video cameras controlled through 2 MX control units and Nexus software. The motion capture and video data are spatiotemporally synchronized. For foot presure data recording, the laboratory uses Tekscan F-scan insole foot pressure sensors and software. This data is recorded simultaneously and synchronized with motion capture during post processing. This equipment is made available through John Challis and the Biomechanics Laboratory.

Each video includes a single subject performance displaying video, mocap based pose and measured foot pressures, as well as stability metrics calculated from the ground truth data.

Sample of video from select subjects in the dataset.

Each video includes a single subject performance displaying the source video with pose data as well as the the predicted pressures from our KNN, PressNET, and PressNet-Simple methods and their associated absolute errors from ground truth.

Application of dataset using open source software for Maya, based on SMPL originally created by Michael J. Black (Director - Perceiving Systems, Max Planck Institute for Intelligent Systems, Tubingen Campus)