When Virtual Reality Meets Internet of Things in the Gym: Enabling - PowerPoint PPT Presentation

When Virtual Reality Meets Internet of Things in the Gym: Enabling Immersive Interactive Machine Exercises Fazlay Rabbi Michigan State University Taiwoo Park* Mi Zhang Biyi Fang Fazlay Rabbi* Youngki Lee (MSU) (MSU) (MSU) (SMU) (MSU) Ubicomp’18 Oct 11 th , 2018 1 * Authors contributed equally

We are living in the era of VR and IoT Oclus Rift Gear VR HTC Vive Virtual Reality IoT We envision that VR and IoT will revolutionize the personal fitness experience. 2

Motivation • Workout in the gym has become part of our lifestyle . • Working out on the machines in gym could make exercisers feel easily bored . • Novice user hardly aware of using right set of muscles , maintaining proper speed etc. • Eventually exercisers lose their motivation and interests . 3

Limitations of Existing Fitness Tracking System Armband/Wristband Smartphone Waist/Chest belt [Muehlbauer et al. 2011] [Chang et al. 2007] [Morris et al. 2014] [Chang et al. 2007] [Pernek et al. 2013] [Velloso et al. 2007] [Mortazavi et al. 2014] • Human-wearable approach is inconvenient for machine exercises. • Unable to track lower body exercise (such as – Seated Abs) 5

Design Goals Machine-attachable sensing device. • Universal sensing platform. • Real-time high accuracy machine exercise information tracking. • Immersive fitness training experience. • 5

JARVIS (Backend) SensorTag Exercise Machine 6

JARVIS (Frontend) VR HMD Fully Immersive Virtual Exercise Assistant 7

Technical Challenges Recognizing the type of machine exercises and track exercise progress with low • latency and high accuracy. Reducing the computational load of the VR rendering to reduce energy • consumption and heat generation. 8

Low latency and high accuracy machine exercises tracking Pulldown Seated Abs 9

Reducing Computational Load 13.8K triangles 30.1K triangles 155.8K triangles 10

Experimental Setup Participants 15 participants. • 11 male and 4 female • 5400 total repetitions. • SensorTag was placed • in two different location. 17

Performance of Repetition Segmentation & Counting • Performance of Repetition Segmentation – Miss Rate (MSR) – Merge Rate (MGR) – Fragmentation Rate (FR) E01 E02 E03 E04 E05 E06 E07 E08 E09 E10 E11 E12 MSR (%) 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 MGR (%) 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.00 0.00 0.00 0.00 FR (%) 0.11 0.11 0.11 0.00 0.33 0.00 0.00 0.22 0.00 0.33 0.00 0.00 • Performance of Repetition Counting § Counting accuracy 97.67% § Within ±1 accuracy 99.81% § Within ±2 accuracy 100% 15

Performance of Exercise Type Recognition • Impact of sensor placement location and session-wise voting: Without Session-wise Voting With Session-wise Voting Location Precision Recall Accuracy F-measure Precision Recall Accuracy F-measure All location 0.8325 0.8266 82.67% 0.8295 0.8970 0.8887 88.88% 0.8930 Location ‘L’ only 0.8102 0.8081 80.82% 0.8091 0.8854 0.8809 88.10% 0.8831 Location ‘S’ only 0.8114 0.8121 81.22% 0.8117 0.8829 0.8815 88.17% 0.8822 Best location 0.9432 0.9430 94.30% 0.9431 0.9911 0.9907 99.08% 0.9909 15

Performance of Exercise Type Recognition • Impact of Number of Features: 99.08% 98% 15

System Performance • Real-time Performance and Power Consumption: Component Processing Time (ms) Repetition Segmentation 0.010 Feature Extraction 0.011 Exercise Type Classification 0.073 Total 0.094 Device Component Current (mA) Power (mW) BLE Communication 61.46 245.7 Processing Backend 17.65 70.57 Low Quality 800.3 3198 Smartphone Mixed Quality 815.6 3259 VR Frontend High Quality 867.9 3468 SensorTag 10 Hz Data Transmission 3.967 11.90 15

Case Study : Experimental Setup • Questionnaire survey, EMG signal study and interview. • Target exercise: Seated Abs (10 participants) • Trigno Wireless sEMG system 15

Case Study : sEMG Signal Analysis • EMG activity: Without VR and With VR in terms of Mean normalized EMG values for RMS values of sEMG signals 4 muscle group 15

Case Study : Survey Analysis • User experience questionnaire survey: IMI survey results for 3 categories 15

Conclusions • JARVIS combines miniature IoT sensing device and a mobile VR headset to enable immersive and interactive gym exercise experience. • With JARVIS, we envision a series of usability and and effectiveness studies will follow up towards realization of effective and practical ubiquitous VR exercise system. 21

Thank You Fazlay Rabbi Michigan State University rabbimd@egr.msu.edu 22