RF-Based 3D Skeletons


Mingmin Zhao      Yonglong Tian      Hang Zhao      Mohammad Abu Alsheikh      Tianhong Li
Rumen Hristov      Zachary Kabelac      Dina Katabi     Antonio Torralba

Massachusetts Institute of Technology


Overview:


RF-Pose3D is the first system that infers 3D human skeletons from RF signals. It requires no sensors on the body, and works with multiple people and across walls and occlusions. Further, it generates dynamic skeletons that follow the people as they move, walk or sit. As such, RF-Pose3D provides a significant leap in RF-based sensing and enables new applications in gaming, healthcare, and smart homes.

RF-Pose3D is based on a novel convolutional neural network (CNN) architecture that performs high-dimensional convolutions by decomposing them into low-dimensional operations. This property allows the network to efficiently condense the spatio-temporal information in RF signals. The network first zooms in on the individuals in the scene, and crops the RF signals reflected off each person. For each individual, it localizes and tracks their body parts -- head, shoulders, arms, wrists, hip, knees, and feet.


Open Positions:

We have openings for postdocs and interns! If you are interested in working on exciting projects like RF-Pose, please send an email along with your resume to Prof. Dina Katabi at dk@mit.edu.


Video:


Coming soon.


Paper:


RF-Based 3D Skeletons
Mingmin Zhao, Yonglong Tian, Hang Zhao, Tianhong Li, Mohammad Abu Alsheikh, Rumen Hristov, Zachary Kabelac, Dina Katabi, Antonio Torralba
ACM SIGCOMM, 2018
[PDF]


Talk:


Coming soon.


Also check out:


Through-Wall Human Pose Estimation using Radio Signals
M. Zhao, T. Li, M. Alsheikh, Y. Tian, H. Zhao, A. Torralba and D. Katabi
Computer Vision and Pattern Recognition (CVPR), 2018

Learning Sleep Stages from Radio Signals: A Conditional Adversarial Architecture
M. Zhao, S. Yue, D. Katabi, T. Jaakkola and M. Bianchi
International Conference on Machine Learning (ICML), 2017

Emotion Recognition using Wireless Signals
M. Zhao, F. Adib and D. Katabi
ACM International Conference on Mobile Computing and Networking (MobiCom), 2016