Detecting State Changes of Indoor Everyday Objects using Wi-Fi - - PowerPoint PPT Presentation

Detecting State Changes of Indoor Everyday Objects using Wi-Fi Channel State Information

• Sep. 12th, 2017

Kazuya Ohara*, Takuya Maekawa, and Yasuyuki Matsushita Osaka University, Japan

1

2

Background

Detecting State Changes of Indoor Everyday Objects (e.g. Open/Close door or window)

• Home automation (adaptive HVAC control)
• Monitoring an independently living elderly person

Application

3

Background

Detecting State Changes of Indoor Everyday Objects (e.g. Open/Close door or window)

• Home automation (adaptive HVAC control)
• Monitoring an independently living elderly person

Application

－Usual Method－  Attach sensors to each object  High deployment and maintenance costs  Use camera images  Invasion of privacy

4

Background

Detecting State Changes of Indoor Everyday Objects (e.g. Open/Close door or window)

• Home automation (adaptive HVAC control)
• Monitoring an independently living elderly person

Application

－Usual Method－  Attach sensors to each object  High deployment and maintenance costs  Use camera images  Invasion of privacy

• The path of Wi-Fi signal is greatly affected by state changes of the indoor everyday objects.
• Wi-Fi communications are used widely in indoor environments.

Wi-Fi signal door PC AP

Wi-Fi signal

5

Purpose

closed

• pened

OPEN transmitter receiver

• Estimate states of indoor everyday objects from propagation information of Wi-Fi signals

based on machine learning techniques. Purpose

• Channel State Information (CSI)
• including the various effects such as

reflection and path loss of signals

• high dimensional complex matrix

Propagation Information  open  close  opened  closed event state

• pen

door closed

• pened

time close Estimating for each time slice

6

Channel State Information

⋮ subcarrier subcarrier

• MIMO (Multiple Input Multiple Output)

using multiple antennas for transmitting and receiving signals

• OFDM (Orthogonal Frequency Division Multiplexing)

using subcarriers whose frequencies are different from each other

7

Channel State Information

⋮

1 Im Re : complex value : amplitude change : phase change

∠

• : # of transmit antennas

: # of receive antennas : # of subcarriers

CSI is obtained at each antenna and subcarrier

amplitude and phase change by reflection

8

Problem

CSI data for each element

CSI

PCA Low dimensional data

Wi-Fi signals are very noisy Noise reduction CSI is complex information including reflection Difficult to design feature Existing approach Noise reduction: Aggregate CSI data and reduce the dimensionality using PCA Feature design: Researchers design manually (waveform, frequency of data)

9

Problem

Activity of one person (e.g. walking, falling ) Existing approach Noise reduction: Aggregate CSI data and reduce the dimensionality using PCA Feature design: Researchers design manually (waveform, frequency of data) Wi-Fi signals are very noisy Noise reduction CSI is complex information including reflection Difficult to design feature

10

Problem

States of multiple indoor objects Mixed effects by objects Noise reduction: Aggregate CSI data and reduce the dimensionality using PCA Feature design: Researchers design manually (waveform, frequency of data) Existing approach Wi-Fi signals are very noisy Noise reduction CSI is complex information including reflection Difficult to design feature separate effects caused by objects

11

Approach

• Noise reduction & Separation: Independent Component Analysis (ICA)
• Feature design: extracted automatically using Convolutional Neural Network (CNN)

12

Approach

• Feature design: extracted automatically using Convolutional Neural Network (CNN)

Blind source separation Aloha Hello ICA Aloha Hello （noise）

Independent source signals Mixed signals Decomposed signals

Aloha Hello

• Noise reduction & Separation: Independent Component Analysis (ICA)

13

Approach

• Feature design: extracted automatically using Convolutional Neural Network (CNN)

ICA

from door from window

(noise)

Signals reflected by objects are independently

• Noise reduction & Separation: Independent Component Analysis (ICA)

14

Approach

• Feature design: extracted automatically using Convolutional Neural Network (CNN)

ICA

from door from window

(noise)

Signals reflected by objects are independently

elements of CSI time

• CSI
• Noise reduction & Separation: Independent Component Analysis (ICA)

15

Overview of Proposed Method

CSI time-series data

time

ICA DNN HMM

• pen

close

• pened

closed Separating effects Feature extraction for each time window Smoothing estimates DNN

elements of CSI

time time

feature

time

estimated state States estimated by DNN include noise. Using HMM, the estimates can be smoothed.

estimated by DNN estimated by HMM time

estimated state

16

Overview of Proposed Method

CSI time-series data

time

ICA DNN HMM

• pen

close

• pened

closed Separating effects ICA DNN HMM

• pen

close

• pened

closed Separating effects

for window for door

ICA DNN HMM

• pen

close

17

ICA

||h11|| ||h12||

Tx 1 – Rx 1 Tx 1 – Rx 2

80 40

time [sec.]

• pen

close

• pen

close Amplitude of CSI when door open/close

Component effected by door noise

80 40

time [sec.]

• pen

close

• pen

close

ICA

Decomposed signals

ICA decomposes amplitude and phase of CSI for each subcarrier.

The number of Tx-Rx antenna pairs

Can not find apparent changes in the signals when events occurred First component captures signal changes when events occurred

18

ICA

ICA

unmixing matrix for door

ICA

Tx 1 – Rx 1 Tx 1 – Rx 2 amplitude amplitude unmixing matrix for window window event window event door event door event

training data

19

Overview of Proposed Method

CSI time-series data

time

ICA DNN HMM

• pen

close

• pened

closed Separating effects Feature extraction for each time window Smoothing estimates DNN

elements of CSI

time time

feature

time

estimated state States estimated by DNN include noise. Using HMM, the estimates can be smoothed.

estimated by DNN estimated by HMM time

estimated state

20

DNN

Convolutional layer LSTM

CSI data

time

extract meaningful features from matrix of CSI data Aggregate features each time window

• Amplitude
• Phase
• Decomposed amplitude
• Decomposed phase

21

DNN

Convolutional layer LSTM

CSI data

time

extract meaningful features from matrix of CSI data Aggregate features each time window

• Amplitude
• Phase
• Decomposed amplitude
• Decomposed phase

Input to HMM

22

Overview of Proposed Method

CSI time-series data

time

ICA DNN HMM

• pen

close

• pened

closed Separating effects Feature extraction for each time window Smoothing estimates DNN

elements of CSI

time time

feature

time

estimated state States estimated by DNN include noise. Using HMM, the estimates can be smoothed.

estimated by DNN estimated by HMM time

estimated state

23

HMM

Output of DNN for each time window

DNN (door)

Left-to-Right HMM

• pen

close

• pened

closed closed

• pen
• pened

closed→open→opened→・・・ Knowledge of objects

• pen

24

HMM

ground truth estimation

closed

• pen
• pened

close

Proposed (HMM estimation w/ grammar)

closed

• pen
• pened

close

HMM estimation w/o knowledge

time [sec] 240 120 60 180 closed

• pen
• pened

close

DNN estimation

25

Evaluation dataset

AP PC window 1 shade 1 door screen 10.2 m 7.2 m AP PC door 7.5 m 3.5 m AP PC 9.5 m 3.5 m

environment 1 environment 2 environment 3

refrigerator

• Each event of object occurred in an arbitrary order

150 sessions (train: 90%, test: 10%)

• Dimension of CSI:
• , CSI was obtained at the rate of 1000Hz.
• We estimate states of objects every 0.1 sec.

window 2 window window shade 2 shade shade cabinet 1 cabinet 2 cabinet 3 cabinet 1 cabinet 2 cabinet 3

26

Evaluation dataset

AP PC window 1 shade 1 door screen 10.2 m 7.2 m AP PC door 7.5 m 3.5 m AP PC 9.5 m 3.5 m

environment 1 environment 2 environment 3

refrigerator

• Each event of object occurred in an arbitrary order

150 sessions (train: 90%, test: 10%)

• Dimension of CSI:
• , CSI was obtained at the rate of 1000Hz.
• We estimate states of objects every 0.1 sec.

window 2 window window shade 2 shade shade cabinet 1 cabinet 2 cabinet 3 cabinet 1 cabinet 2 cabinet 3 window screen shade door

27

Result for each objects

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

door window1 window2 shade1 shade2 screen door window shade cabinet1 cabinet2 cabinet3 window shade cabinet1 cabinet2 cabinet3 refrigerator environment 1 environment 2 environment 3

Averaged F-measure

AP AP PC PC window 1 shade 1 door screen environment 1 window 2 shade 2 AP AP PC PC environment 3 refrigerator window shade cabinet 1 cabinet 2 cabinet 3 AP AP PC PC door environment 2 window shade cabinet 1 cabinet 2 cabinet 3

28

Result for each event/state

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 door window1 window2 shade1 shade2 screen door window shade cabinet1 cabinet2 cabinet3 window shade cabinet1 cabinet2 cabinet3 refrigerator environment 1 environment 2 environment 3

F-measure

• pen

close

• pened

closed

29

Result for each event/state

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 door window1 window2 shade1 shade2 screen door window shade cabinet1 cabinet2 cabinet3 window shade cabinet1 cabinet2 cabinet3 refrigerator environment 1 environment 2 environment 3

F-measure

• pen

close

• pened

closed

closed

• pen
• pened

close

Proposed

time [sec] 240 120 60 180

50,000 100,000 150,000 200,000

• pen

close

• pened closed

The number of samples

10 times ground truth estimation

30

Result for each event/state

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 door window1 window2 shade1 shade2 screen door window shade cabinet1 cabinet2 cabinet3 window shade cabinet1 cabinet2 cabinet3 refrigerator environment 1 environment 2 environment 3

F-measure

• pen

close

• pened

closed

31

Comparison with Baselines

Averaged F-measure 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 environment 1 environment 2 environment 3 Proposed w/o ICA w/o grammar w/o HMM RF

• Proposed: our proposed method
• w/o ICA: don’t use decomposed

amplitude and decomposed phase

• w/o knowledge: don’t use

knowledge of objects

• w/o HMM: estimate states of
• bjects by DNN
• RF: estimate states by Random

Forests using CSI denoised by PCA 5.5% 5.2% 1.3% 21.5% 27.2% 20.4%

knowledge

32

Result of Long-term experiment

AP PC door 7.5 m 3.5 m

environment 2

window shade cabinet 1 cabinet 2 cabinet 3

• Collect data for 10 days in environment 2 (20 sessions/day)
• train: first 3 days (60 sessions), test: last 7 days (140 sessions)
• Fine-tuning DNN by recognized result as pseudo ground truth for each day

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1 2 3 4 5 6 7 w/o fine-tuning w/ fine-tuning Averaged F-measure days

33

Amount of training data

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 2 3 4 5 6 7 AP PC door 7.5 m 3.5 m

environment 2

window shade cabinet 1 cabinet 2 cabinet 3 Averaged F-measure days

• Change amount of training data from 1 day to 3 days

i.e. train: 20 sessions, 40 sessions, 60 sessions

train: 1 day train: 2 days train: 3 days

future task: method that does not require vast training data

34

Conclusion

• We proposed a method for detecting events and states of indoor everyday objects

such as doors and windows using Wi-Fi CSI

• We designed a novel processing pipeline based on ICA, DNN, and HMM with a

grammar based on knowledge of an object.

• We investigated the effectiveness of our method using real data. And we confirmed

that our approach significantly outperformed a classic machine learning-based approach.

• However, because our method relies on deep learning, much training data results in

poor recognition accuracy.

• Cope with this problem is one of our important future tasks.