Inference Using Wrist-Based Motion Sensors Revisited Raveen - - PowerPoint PPT Presentation

deWristified: Handwriting Inference Using Wrist-Based Motion Sensors Revisited

Raveen Wijewickrama

raveen.wijewickrama@utsa.edu

Anindya Maiti

a.maiti@ieee.org

Murtuza Jadliwala

murtuza.jadliwala@utsa.edu

University of Texas at San Antonio

Wrist Wearables

• Extends the functionality of traditional wristwatches beyond

timekeeping.

• Captures rich contextual information about the wearer.
• Enables several novel context-based applications.

2019-05-16 SPriTELab @ UTSA 2

Motion Sensors

• Two main types of motion or inertial sensors:
• Accelerometer: records device acceleration.
• Gyroscope: records device angular rotation.
• Accessing motion sensors on wearable devices:
• All applications have access to motion sensors by default (also referred to as

zero-permission sensors) on most wearable OSs.

• Applications’ access to motion sensors cannot be regulated on most wearable OSs –

we can’t turn them off!

• Can an adversary take advantage of motion sensor data from a

wrist-wearable device to infer private information inputted by the user’s device-wearing hand?

2019-05-16 SPriTELab @ UTSA 3

Inferring Private User Inputs (Using Wrist Wearables)

2019-05-16 4 SPriTELab @ UTSA

State-of-the-Art in Handwriting Recognition (Using Wrist Wearables)

2019-05-16 5 SPriTELab @ UTSA

Airwriting (Amma et al.) Whiteboard writing (Arduser et al.) Finger writing (Xu et al.) Pen(cil) writing (Xia et al.)

Adversary Model

• Adversary has knowledge of the type of handwriting.
• Adversary is able to record data from the target smartwatch’s

accelerometer and gyroscope sensors.

• Could employ a Trojan app for this!
• Adversary’s Goal: To infer handwritten information using target

user’s smartwatch sensors.

2019-05-16 SPriTELab @ UTSA 6

Limitations of Earlier Handwriting Recognition Studies (Using Wrist Wearables)

• Airwriting (Amma et al.)
• Custom-designed hand glove with very

high precision sensors.

• Our adversary relies on target user’s

smartwatch or fitness band.

• Only uppercase words.
• Whiteboard writing (Arduser et al.)
• Not generalized (training and testing

data not from different participants).

• Only uppercase alphabets.
• No handwriting activity detection.
• Finger writing (Xu et al.)
• Use of Shimmer, a specialized sensing

device intended for lab studies.

• Not generalized (training and testing

data not from different participants).

• Pen(cil) writing (Xia et al.)
• Only lowercase alphabets.
• Controlled data collection.
• No handwriting activity detection.

2019-05-16 SPriTELab @ UTSA 7

Our Research

• How practical is handwriting

inference when

• Using consumer-grade wrist

wearables,

• Using generalized training and

testing,

• Writing in a uncontrolled and

unconstrained manner, and

• Both upper and lowercase

alphabets are modeled ?

2019-05-16 SPriTELab @ UTSA 8

New Uncontrolled and Unconstrained Writing Data Existing Models

Handwriting Inference Framework

2019-05-16 9 SPriTELab @ UTSA

Experimental Setup

• 28 participants for the four writing scenarios.
• 18 to 30 years of age
• 13 male, 15 female
• Two different wrist-wearables.
• Sony Smartwatch 3, LG Watch Urbane
• Accelerometer and gyroscope recorded at 200Hz.
• Participants provided with appropriate writing apparatus.

2019-05-16 SPriTELab @ UTSA 10

Writing Tasks (In-Lab)

• Alphabets.
• Individual alphabets one at a time.
• Covered all 26 English alphabets in random order.
• Each alphabet was written 10 times.
• Both upper and lower cases.
• Words.
• 4-8 alphabet words, from a vocabulary (Goldhahn et al. 2012).
• Each participant wrote 20 words, in both upper and lower cases.
• Sentence.
• "the five boxing wizards jump quickly" in both upper and lower cases.

2019-05-16 SPriTELab @ UTSA 11

Writing Activity Recognition (Out of Lab)

• 2 participants.
• Wore a smartwatch for an entire day.
• Performed the four writing scenarios at random times.
• Adversary’s Goal: To infer handwriting activity first, and then

classify the handwritten text.

2019-05-16 SPriTELab @ UTSA 12

Replicated Inference Frameworks

• Airwriting
• Hidden Markov Model (HMM)
• Whiteboard writing
• Dynamic Time Warping (DTW)
• Finger writing
• Naive Bayes, Logistic Regression and Decision Tree classifiers
• Pen(cil) writing
• Random Forest classifier

2019-05-16 SPriTELab @ UTSA 13

Personalized Inference Accuracy

2019-05-16 SPriTELab @ UTSA 14

Writing Activity Detection: 56% recall and 57% precision for air and finger writing 39% recall and 47% precision for pencil writing 23% recall and 34% precision for whiteboard writing

Personalized Inference Accuracy (Whiteboard Writing)

2019-05-16 SPriTELab @ UTSA 15

Lowercase Uppercase

Generalized Inference Accuracy

2019-05-16 SPriTELab @ UTSA 16

Writing Activity Detection: 35-40% recall for airwriting, whiteboard writing and pencil writing Only 8% recall for finger writing

Factors Affecting Inference Accuracy

• Number of Strokes.

2019-05-16 SPriTELab @ UTSA 17

Factors Affecting Inference Accuracy

2019-05-16 SPriTELab @ UTSA 18

Number of strokes for the same letter for different participants (lowercase). Number of strokes for the same letter for different participants (uppercase).

Factors Affecting Inference Accuracy

2019-05-16 SPriTELab @ UTSA 19

Lowercase Uppercase Variance in number of strokes per alphabet per participant, averaged for all participants

Factors Affecting Inference Accuracy

• Number of Strokes.
• Order of Strokes.
• Direction of Strokes.

2019-05-16 SPriTELab @ UTSA 20

Factors Affecting Inference Accuracy

2019-05-16 SPriTELab @ UTSA 21

Factors Affecting Inference Accuracy

• Number of Strokes.
• Order of Strokes.
• Direction of Strokes.
• Uppercase vs Lowercase.
• Specialized Devices.

2019-05-16 SPriTELab @ UTSA 22

Airwriting (Amma et al.)

Conclusion

• We investigated how wrist-wearable based handwriting

inference attacks perform in realistic day-to-day writing situations.

• Such inference attacks are unlikely to pose a substantial threat

to users of current consume-grade smartwatches and fitness bands.

• Primarily due to highly varying nature of handwriting.
• Replicable artifacts: https://sprite.utsa.edu/art/dewristified

2019-05-16 SPriTELab @ UTSA 23