SLIDE 1
Selecting Home Appliances with Smart Glass based on Contextual - - PowerPoint PPT Presentation
Selecting Home Appliances with Smart Glass based on Contextual - - PowerPoint PPT Presentation
Selecting Home Appliances with Smart Glass based on Contextual Information Ubicomp 2016 *Quan Kong(Osaka University) Takuya Maekawa(Osaka University) Taiki Miyanishi(ATR) Takayuki Suyama(ATR) Advanced Telecommunications Research Advanced
SLIDE 2
SLIDE 3
Osaka University Advanced Telecommunications Research Institute International
Approaches for Home Appliances Control
Voice Gesture Wearable Camera IR
Wearable Camera
Shi F., 06
“ Alex, open the curtain on the kitchen’s north wall.”
IR Emitter Ben Z., S UI ’ 14
Amazon echo
SLIDE 4
Osaka University Advanced Telecommunications Research Institute International
Our Approach
“ ON”
+
Living Room Television
Home Network
・Activity ・Indoor position
SLIDE 5
Osaka University Advanced Telecommunications Research Institute International
Feature of Our Approach – Context-aware appliance selection
Google Glass Camera
・Orientation ・Accelerator ・Light sensor
…
Context Information Indoor positions Activities
non-paramet ric unsupervised learning
Why cont ext informat ion ?
Related to the home appliances Distinguish between different appliances
SLIDE 6
Osaka University Advanced Telecommunications Research Institute International
System Overview- Sensors Used in Our System
Screen Light sensor Orientation
Google Glass Smartphone
Wi-Fi Microphone Accelerator Camera Appliance Image (First person view) Camera Activity Information Microphone Accelerator Light sensor Wi-Fi Indoor position Information Orientation Head direction
SLIDE 7
Osaka University Advanced Telecommunications Research Institute International
Update the model by using the collected history daily data Use the trained appliance selection model in the daily life Select the prepared name of appliance in glass application
Stand in front of the appliance and take a 10 seconds video
System Overview- Initialization and Model Update
1 2
Images extracted from the video used as the training data for initializing
3 4 5
wrong estimation is corrected by the user and used as training data
Initialization
SLIDE 8
Osaka University Advanced Telecommunications Research Institute International
Proposed Method
Extract the attention time’s image feature and estimate the activity & position (IGMM)
Detect the user’s attention using orientation data
1 2
Extracted above information as the input of appliance selection model (MKL)
3
Appliance Image Sensor Data Position Activity
+
Deep Convolut ional Neural Net work
Extract CNN fc6 CNN feature fc6:4096 dimensional Feature Extraction Train(ed) Activity Model Train(ed) Positional Model
Train Dat a (t est dat a) Feat ure Dat a
0.981 Act 1 0.765 Act 2 0.543 Act 3
… …
Multiple Kernel Learning
Classification
Estimated Appliance
Context feature
Clust ering
0.923 Pos1 0.865 Pos2 0.443 Pos3
… …
Attention Detection
SLIDE 9
Osaka University Advanced Telecommunications Research Institute International
Proposed Method – Image feature extraction with DCNN
image feature
CaffeNet Reference Model: ILSVR-2012
SLIDE 10
Osaka University Advanced Telecommunications Research Institute International
Proposed Method- Unsupervised activity recognition and indoor positioning
Wi-Fi
- light
- Acc
- microphone
Position IGMM Activity IGMM
Feature Extraction
Test data
𝐸 : the invers of the distance between the test data and each cluster 𝐸 𝐸 Bedroom Kitchen Cook Sleep
- Wi-Fi
- microphone
- light
- Acc
Feat ure ext ract ion of IGMM input
Accelerator Microphone Wi-Fi Light sensor 3-axis combination signal Average MFCC components Average of illumination Signal strength values
Learning Act ivit y and Posit ion Model ・Use non-parametric learning approach IGMM for activity and position clustering
SLIDE 11
Osaka University Advanced Telecommunications Research Institute International
Proposed Method –Appliance selection using MKL
Camera image Wi-Fi
- light
- Acc
- microphone
Selected Appliance Position IGMM Activity IGMM
Deep convolutional neural network Multiple kernel learning
Image features context features
A linear combinat ion of mult iple base kernels for image and cont ext feat ure
describe a different property of the data with multiple kernels
Mult iple Kernel Learning
𝒍𝒋𝒏𝒉,∗: 𝑞𝑝𝑚𝑧𝑜𝑝𝑛𝑗𝑏𝑚 𝑙𝑓𝑠𝑜𝑓𝑚 𝑔𝑝𝑠 𝑗𝑛𝑏𝑓 𝒍𝒅𝒑𝒐𝒖𝒇𝒚𝒖,∗: 𝑠𝑏𝑒𝑗𝑏𝑚 𝑐𝑏𝑡𝑗𝑡 𝑔𝑣𝑜𝑑𝑢𝑗𝑝𝑜（𝑔𝑝𝑠 𝑑𝑝𝑜𝑢𝑓𝑦𝑢）
Decision Function : f𝑦∗ 𝑏 𝑓𝑙,∗ 𝑓𝑙,∗ 𝑐
SLIDE 12
Osaka University Advanced Telecommunications Research Institute International
Device for dat a collect ion
- Google Glass, Nexus 5 (in pocket)
- Sampling rate : 30Hz
Semi-naturalistic collection protocol
- Activities follow the instruction
- 3 users X 10 sessions activities
prepare meals eat meals wash dishes watch TV
…
sleep
Random
Activities
floor plan and appliances
toilet faucet bedroom air conditioner bedroom lighting lounge lighting TV lounge air conditioner kitchen faucet kitchen curtain lounge curtain kitchen lighting drawer front door fan
Evaluation – Data set
SLIDE 13
Osaka University Advanced Telecommunications Research Institute International
Evaluation Result - Leave-one-session out cross validat ion
Proposed: activity + position + camera Proposed w/o pos: activity+ camera Proposed w/o act: position + camera Proposed w/ cam:
- nly camera
SVM w/ cam:
- nly camera
SVM all:
activity + position + camera
Effect of context
F-measure Recall Precision
10%
0.845 0.813 0.857 0.928 0.894 0.955 0.844 0.812 0.862 0.897 0.878 0.936 0.844 0.812 0.859 0.912 0.886 0.945 SVM all SVM w/ cam Proposed w/ cam Proposed w/ o act Proposed w/ o pos Proposed
SLIDE 14
Osaka University Advanced Telecommunications Research Institute International
Evaluation Result – Confusion Matrix
Visual confusion matrix of Proposed w/ cam Visual confusion matrix of Proposed
- air conditioner and lighting were relatively poor
- can’ t distinguish between kitchen lighting and
bedroom lighting
- drawer performed not well
- air conditioner and lighting were increased
about 14%on average of F-measure
- F-measure improved by about 10% on total
average
SLIDE 15
Osaka University Advanced Telecommunications Research Institute International
0.4 0.5 0.6 0.7 0.8 0.9 1 2 4 6 8 10 12 Proposed Proposed (reuse) Proposed (imgaenet )
Evaluation Result –Transition of Average F-measures
sessions F-measure
Reusing ot her users’ dat a
Appliance Image Sensor Data Position Activity
+
Appliance Image Sensor Data Position Activity
+
Appliance Image Sensor Data Position Activity
+
User1 User2 User3
- collect online
images for each category
- find top-k similar
images for each appliance
Ut ilizing online image dat abase
32% 28%
SLIDE 16