When Virtual Reality Meets Internet of Things in the Gym: Enabling - - PowerPoint PPT Presentation

Fazlay Rabbi

Michigan State University

When Virtual Reality Meets Internet of Things in the Gym: Enabling Immersive Interactive Machine Exercises

Mi Zhang

(MSU)

Biyi Fang

(MSU)

Fazlay Rabbi*

(MSU)

1

Ubicomp’18 Oct 11th, 2018 Taiwoo Park*

(MSU)

Youngki Lee

(SMU)

* Authors contributed equally

2

We are living in the era of VR and IoT

Oclus Rift IoT Virtual Reality Gear VR HTC Vive

We envision that VR and IoT will revolutionize the personal fitness experience.

3

Motivation

• Workout in the gym has become part of our lifestyle.
• Working out on the machines in gym could make exercisers feel easily bored.
• Novice user hardly aware of using right set of muscles, maintaining proper

speed etc.

• Eventually exercisers lose their motivation and interests.

Limitations of Existing Fitness Tracking System

Armband/Wristband

[Morris et al. 2014] [Chang et al. 2007] [Mortazavi et al. 2014]

Smartphone

[Muehlbauer et al. 2011] [Pernek et al. 2013]

Waist/Chest belt

[Chang et al. 2007] [Velloso et al. 2007]

5

• Human-wearable approach is inconvenient for machine exercises.
• Unable to track lower body exercise (such as – Seated Abs)

5

Design Goals

• Machine-attachable sensing device.
• Universal sensing platform.
• Real-time high accuracy machine exercise information tracking.
• Immersive fitness training experience.

6

JARVIS (Backend)

SensorTag Exercise Machine

7

JARVIS (Frontend)

Fully Immersive Virtual Exercise Assistant

VR HMD

8

Technical Challenges

• Recognizing the type of machine exercises and track exercise progress with low

latency and high accuracy.

• Reducing the computational load of the VR rendering to reduce energy

consumption and heat generation.

9

Low latency and high accuracy machine exercises tracking

Pulldown Seated Abs

10

Reducing Computational Load

13.8K triangles 30.1K triangles 155.8K triangles

Participants

• 15 participants.
• 11 male and 4 female
• 5400 total repetitions.
• SensorTag was placed

in two different location.

Experimental Setup

17

Performance of Repetition Segmentation & Counting

15

• Performance of Repetition Segmentation

– Miss Rate (MSR) – Merge Rate (MGR) – Fragmentation Rate (FR)

• Performance of Repetition Counting

§ Counting accuracy 97.67% § Within ±1 accuracy 99.81% § Within ±2 accuracy 100%

E01 E02 E03 E04 E05 E06 E07 E08 E09 E10 E11 E12 MSR (%) 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 MGR (%) 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.00 0.00 0.00 0.00 FR (%) 0.11 0.11 0.11 0.00 0.33 0.00 0.00 0.22 0.00 0.33 0.00 0.00

Performance of Exercise Type Recognition

15

• Impact of sensor placement location and session-wise voting:

Location Without Session-wise Voting With Session-wise Voting Precision Recall Accuracy F-measure Precision Recall Accuracy F-measure All location 0.8325 0.8266 82.67% 0.8295 0.8970 0.8887 88.88% 0.8930 Location ‘L’ only 0.8102 0.8081 80.82% 0.8091 0.8854 0.8809 88.10% 0.8831 Location ‘S’ only 0.8114 0.8121 81.22% 0.8117 0.8829 0.8815 88.17% 0.8822 Best location 0.9432 0.9430 94.30% 0.9431 0.9911 0.9907 99.08% 0.9909

Performance of Exercise Type Recognition

15

• Impact of Number of Features:

99.08% 98%

System Performance

15

• Real-time Performance and Power Consumption:

Component Processing Time (ms) Repetition Segmentation 0.010 Feature Extraction 0.011 Exercise Type Classification 0.073 Total 0.094 Device Component Current (mA) Power (mW) Smartphone BLE Communication 61.46 245.7 Processing Backend 17.65 70.57 VR Frontend Low Quality 800.3 3198 Mixed Quality 815.6 3259 High Quality 867.9 3468 SensorTag 10 Hz Data Transmission 3.967 11.90

Case Study : Experimental Setup

15

• Questionnaire survey, EMG signal study and interview.
• Target exercise: Seated Abs (10 participants)
• Trigno Wireless sEMG system

Case Study : sEMG Signal Analysis

15

• EMG activity:

Without VR and With VR in terms of RMS values of sEMG signals Mean normalized EMG values for 4 muscle group

Case Study : Survey Analysis

15

• User experience questionnaire survey:

IMI survey results for 3 categories

• JARVIS combines miniature IoT sensing device and a mobile VR headset to

enable immersive and interactive gym exercise experience.

• With JARVIS, we envision a series of usability and and effectiveness studies

will follow up towards realization of effective and practical ubiquitous VR exercise system.

Conclusions

21

Thank You

Fazlay Rabbi Michigan State University rabbimd@egr.msu.edu

22