T T TH -Typing on Your TeetH: Tongue-Teeth Localization for - - PowerPoint PPT Presentation

▶
t t
SMART_READER_LITE
LIVE PREVIEW

T T TH -Typing on Your TeetH: Tongue-Teeth Localization for - - PowerPoint PPT Presentation

Jan 26, 2024 267 likes •725 views

T T TH -Typing on Your TeetH: Tongue-Teeth Localization for Human-Computer In Interface Phuc Nguyen, Nam Bui, Anh Nguyen, Hoang Truong, Abhijit Suresh, Matthew Whitlock, Duy Pham, Thang Dinh, and Tam Vu Mobile and Networked Systems Lab

slide-1
SLIDE 1

T T TH-Typing on Your TeetH: Tongue-Teeth

Localization for Human-Computer In Interface

Phuc Nguyen, Nam Bui, Anh Nguyen, Hoang Truong, Abhijit Suresh, Matthew Whitlock, Duy Pham, Thang Dinh, and Tam Vu

Mobile and Networked Systems Lab Department of Computer Science, University of Colorado Department of Computer Science, Virginia Commonwealth University

ACM MobiSys 2018

slide-2
SLIDE 2

Lower Jaw Upper Jaw

2

ABC

1 4

GHI

5

JKL

3

DEF

8

TUV

6

MNO

7

PQRS

9

WXYZ +

* #

slide-3
SLIDE 3

It is challenging for ALS patients to interact with computing devices

slide-4
SLIDE 4

A HAND-FREE INTERFACE FOR FACTORY WORKER TURNING MUSIC SHEETS FOR MUSICIANS CONTROLLING PHONE WHILE DRIVING USED IN TACTICAL SCENARIOS

Other Potential Usages

HIDEN TEXT ENTRY INTERFACE

slide-5
SLIDE 5

TYTH – Typing on Your TeetH

  • Non-invasive,
  • Continuous & long-term use,
  • Socially acceptable
slide-6
SLIDE 6

Experiment: Put your fingers at the back-of-the ear locations. Then, press any teeth using your tongue

slide-7
SLIDE 7

Primary Motor Cortex Cortex Sensorial Cortex Motor

Anatomical and Neurological Analysis

  • f Tongue-Teeth Interaction
slide-8
SLIDE 8

Brain sends commands out

Primary Motor Cortex Cortex Sensorial Cortex Motor

slide-9
SLIDE 9

Anatomical Analysis of Tongue-Teeth Movement

Primary Motor Cortex

EEG EEG EMG

slide-10
SLIDE 10

Tongue is controlled by extrinsic and intrinsic muscles

Genioglossus Styloglossus Dorsal surface

  • f tongue

Hyoglossus Mandible bone

slide-11
SLIDE 11

TongueSee – CHI’14

slide-12
SLIDE 12

It is difficult to make the device socially acceptable

TongueSee – CHI’14

slide-13
SLIDE 13

Tongue is controlled by extrinsic and intrinsic muscles

Genioglossus Styloglossus Dorsal surface

  • f tongue

Hyoglossus Mandible bone

TYTH

Can we capture the tongue movement signal from this location?

slide-14
SLIDE 14

Experimental Validation

slide-15
SLIDE 15

Hardware Design

slide-16
SLIDE 16

Sensing Techniques

EEG Sensing EMG Sensing SKD Sensing

EMG EEG EEG SKD

slide-17
SLIDE 17

Sensing Techniques

EEG Sensing EMG Sensing SKD Sensing

slide-18
SLIDE 18

Sensing for EEG/EMG

COTS Bio ioelectrical Sensing Cir ircuit

  • Programmable Gain
  • Programmable Filters
  • Low-Noise, 8 ADC Channel
  • 16kSample/s
  • uV sensitivity
slide-19
SLIDE 19

Sensing Techniques

EEG Sensing EMG Sensing SKD Sensing

low amplitude low frequency signal

slide-20
SLIDE 20

Skin Surface Deformation Sensing Technique

Cooper (Gold plated) Cooper tape (Gold plated) d Human skin

Capacitance Permitivity Area size Distance b/w 2 plates

Soft material (Silicon Dragon 0-10) Capacitive sensing approach

slide-21
SLIDE 21

Sensing Techniques

EEG Sensing EMG Sensing SKD Sensing

Challenge: These signals are extremely weak (mV/uV)

slide-22
SLIDE 22
  • Software Components to:
  • De-noising the signal
  • Extracting EEG, EMG from bio-electrical sensing data
  • Detecting when the tongue is typing/pressing
  • Classifying where the tongue is taping on
  • Recognizing untrained areas

Challenge: These signals are extremely weak (mV/uV)

slide-23
SLIDE 23

System Overview

TYTH wearable device

Electrical Sensor, Capacitive Sensor 60Hz Notch Filter Band Pass Filter Analog Amplifier BLE Communication

Host Computer Pre-processing

Notch Filtering BP Filtering Low-rank Analysis EEG, EMG, SKD signals

Typing Detector

Tongue movement Detection Tongue Pressing Detection

Wavelet STFT Typing Recognizer

Host Device

Feature Extraction SVM GMM Regression Model Classification Key Mapping Feedback Key Generation Localization

slide-24
SLIDE 24

Low-Rank Analysis

  • Every bio-signal f(x) can be represented as:
  • Or,
  • Hence,

Number of Gabor atoms in a dictionary atoms coefficient

Building dictionaries to extract the main structures of the signal. Each dictionary represents the key structure of each signal type (EEG, EMG)

Please refer our paper for more details

slide-25
SLIDE 25

Low-Rank Analysis

f(x) f(xEEG) f(xEMG) f(xnoise) f(xmain structure) f(xdetail structure )

slide-26
SLIDE 26

Pressing Detection

Detecting the Tongue Movement by identifying the discontinuity of the signal. Detecting the Tongue Pressing based on the presence of the brain signal

Tongue movement Detection

Wavelet

Tongue Pressing Detection

  • How do we detect when a user is tapping?
  • How do we detect when a user is

pressing the teeth?

STFT

slide-27
SLIDE 27

Typing Area Classification: SVM - GMM

SVM

RBF mean vector

Feature Extraction

MFCC features delta double delta

Expectation Maximization Initialization (UBM) Linear Cosine

GMM

Building a classification model to detect the trained typing areas

slide-28
SLIDE 28

Typing Area Localization

42 dimensions GMM output

PCA

3D space coordination

During typing, typing location might not be exactly where it is trained. We build a regression model to detect the untrained areas to recognize these locations

Please refer our paper for more details

slide-29
SLIDE 29

Summary

  • Anatomical and Neurological Analysis
  • Hardware Sensing Design
  • Software Components

Let’s put things together

slide-30
SLIDE 30

TYTH’s Prototype

slide-31
SLIDE 31

TYTH’s Prototype

slide-32
SLIDE 32

TYTH’s Prototype

EEG Sensors EMG Sensors SKD Sensors

slide-33
SLIDE 33

TYTH’s Prototype

slide-34
SLIDE 34

Performance Evaluation

Teeth areas for evaluation

slide-35
SLIDE 35

Typing Detection

slide-36
SLIDE 36

Recognition Performance

Average Accuracy: 88.61%

Ground Truth Classified Area

slide-37
SLIDE 37

User Study

Ease to use TYTH

1 – Extremely Difficult 2 – Difficult 3 – Normal 4 – Easy 5 – Extremely Easy

%

slide-38
SLIDE 38

User Study

What are the speed of typing on the teeth? Typing speed (s)

slide-39
SLIDE 39

TYTH’s Sensing Techniques

EEG Sensing EMG Sensing SKD Sensing

slide-40
SLIDE 40

Conclusions

  • We introduce TYTH-Typing on Your TeetH,
  • A Non-invasive,
  • Continuous and Long-term use
  • and Socially Acceptable

wearable device for Tongue-Teeth Localization Applications.

  • The key contributions include:
  • An analysis of brain, muscle, and skin deformation from behind the ears
  • An algorithm to extract the EEG, EMG signals
  • A novel method to sense a new type of signal, termed SKD signal
  • A ear-mounted wearable prototype
  • An evaluation of the system on 15 subjects
slide-41
SLIDE 41

In progress

  • Miniaturization
  • Remove the impact of talking and body movement artifacts
  • Improve the form factor for better contact quality
slide-42
SLIDE 42

T T TH-Typing on Your TeetH: Tongue-Teeth

Localization for Human-Computer In Interface

Phuc Nguyen, Nam Bui, Anh Nguyen, Hoang Truong, Abhijit Suresh, Matthew Whitlock, Duy Pham, Thang Dinh, and Tam Vu

Mobile and Networked Systems Lab Department of Computer Science, University of Colorado Department of Computer Science, Virginia Commonwealth University

ACM MobiSys 2018

Danke schön !!!

slide-43
SLIDE 43

Thank you

slide-44
SLIDE 44

Related Works

TongueSee – CHI’14 SITA - UIST '12 Tongue-in-Cheek – CHI’15 TongueWise –EMBC’10