SensePresence: Infrastructure-less Occupancy Detection for - - PowerPoint PPT Presentation

SensePresence: Infrastructure-less Occupancy Detection for Opportunistic Sensing Applications

16th IEEE International Conference on Mobile Data Management

Md Abdullah Al Hafiz Khan H M Sajjad Hossain Nirmalya Roy

University of Maryland Baltimore County Mobile, Pervasive & Sensor Computing Lab Universities of Maryland Baltimore County

June 15, 2015

Md Abdullah Al Hafiz Khan (UMBC) SensePresence June 15, 2015 1 / 27

Outline

1

Motivation Class Participation Social Gathering Group member’s participation

2

Solutions

3

Ubiquitous Voice Sensing Ubiquitous Sensing What we have already? Voice Centric Sensing

4

Goals and Challenges

5

Overview of SensePresence

6

Methodology Speaker Counting Algorithm Locomotive Counting

7

Experimental Setup and Results

8

Discussion & Future Work

Md Abdullah Al Hafiz Khan (UMBC) SensePresence June 15, 2015 2 / 27

Which class is interactive?

Helps to solve problems and theories. Helps gain knowledge. Total interactive participants.

Md Abdullah Al Hafiz Khan (UMBC) SensePresence June 15, 2015 3 / 27

How many people are there?

Is the party enjoyable?

Md Abdullah Al Hafiz Khan (UMBC) SensePresence June 15, 2015 4 / 27

Group member’s participation

How many people participate in the meeting? Does all the member participate?

Md Abdullah Al Hafiz Khan (UMBC) SensePresence June 15, 2015 5 / 27

Solutions

People Count!

◮ Which class is Interactive? ⋆ Check how many students ask questions? ◮ Where is the party? ⋆ Find the place where most people speaks. ◮ Is the meeting effective? ⋆ How many members participate?

Microphone + Accelerometer Sensors

Md Abdullah Al Hafiz Khan (UMBC) SensePresence June 15, 2015 6 / 27

Ubiquitous Sensing

Brace Smart Watch Necklace Phone

What are the sensors available today? Which smart devices belongs to all people?

Md Abdullah Al Hafiz Khan (UMBC) SensePresence June 15, 2015 7 / 27

What we have already?

Accelerometer Microphone Gyroscope etc.

Md Abdullah Al Hafiz Khan (UMBC) SensePresence June 15, 2015 8 / 27

Voice Centric Sensing

Speaker Recognition Speaker Counting Speaker Identification Emotion Detection Stress Detection 2 3

What are the different types of application using voice centric sensing? “Blind Speaker clustering”, Iyer, IEEE, ISPACS (2006) “Crowd++: Unsupervised speaker count with smartphones”, Chenren Xu, UbiComp (2013): Static segmentation, controlled scenario where all speakers are active

Md Abdullah Al Hafiz Khan (UMBC) SensePresence June 15, 2015 9 / 27

Goals and Challenges

Challenges Solution No prior knowledge of speakers Best Feature Extraction Background noise Filter Some people might remain silent Other Sensor (Accelerometer) Speech overlap Overlap Detection Privacy concern Use encryption (steganographic,stego)

Md Abdullah Al Hafiz Khan (UMBC) SensePresence June 15, 2015 10 / 27

SensePresence Architecture

Occupancy Context Model (OCM)

People Count

Pitch Estimation Pre-Processing

Node List Speaker Count

Optimum Node Estimate Proximity Signature Collection Feature Extraction Occupancy Estimation Accelerometer Microphone AFP Trigger Sink Sink

n mobile phone

Locomotive Context model (LCM) Acoustic Context Model (ACM) Server-side Architecture

Mobile-side Architecture

Gender Detection MFCC Speaker Estimation Md Abdullah Al Hafiz Khan (UMBC) SensePresence June 15, 2015 11 / 27

Methodology

Acoustic methodology

◮ Create segment from raw audio ◮ Find Male and Female Segments ◮ Audio Processing

Locomotive methodology

◮ Select sensors data based on speaker count and node list ◮ Calculate Magnitude ◮ Detect abrupt changes on the signal Md Abdullah Al Hafiz Khan (UMBC) SensePresence June 15, 2015 12 / 27

Case 1: when people are conversing

Dynamic Segmentation

0.91 0.92 0.93 0.94 0.95 0.96 0.97 0.98 0.99 1 1.01 0.32 0.48 0.64 0.8 0.96 1.12 1.28 1.44 1.6 1.76 1.92 2.08 2.24 2.4 2.56 2.72 2.88 3.04 3.2 3.36

Confidence Scores Segment Lengths

Confidence Score Vs. Segment Length

What is the minimum or maximum segment length? Consider higher confidence score Which segment to choose when multiple segments have same confidence (i.e. 2.72 vs. 3.36 seconds)

Md Abdullah Al Hafiz Khan (UMBC) SensePresence June 15, 2015 13 / 27

Gender Detection

Calculate Pitch Human voice ranges from 50Hz to 450Hz Male pitch falls between 100Hz to 146Hz Female pitch falls between 188Hz to 221Hz. Make Male and Female Segment sets.

Md Abdullah Al Hafiz Khan (UMBC) SensePresence June 15, 2015 14 / 27

Audio Processing

Filter [300 Hz – 4 kHz] Framing Windowing Audio Signal Dynamic Segmentation Frames Hamming window (50% overlapped) Frame length 32 ms Band pass filter (300Hz - 4000Hz)

Md Abdullah Al Hafiz Khan (UMBC) SensePresence June 15, 2015 15 / 27

Mel-frequency cepstral coefficients

DFT Mel-filter Bank Discrete Cosine Transform Delta Energy& Spectrum Output: MFCC Frames

Take Fourier transform Apply triangular mel-filter bank to map the power of the spectrum and take log Apply Discrete cosine transform Amplitude of the spectrum is the MFCC

Md Abdullah Al Hafiz Khan (UMBC) SensePresence June 15, 2015 16 / 27

Segment Sets Sorting

Calculate Intra-frame angles Sort Segments

s1 s2 s3 sn s1 s4 sn S3 Calculate intra-frame cosine angles Take average intra-frame angles Sort segment based on avg. angle

Md Abdullah Al Hafiz Khan (UMBC) SensePresence June 15, 2015 17 / 27

Grouping of Human Speakers based on Proximity S1 S2 S3 S4 S6 S5 Bucket 1 Bucket 2 Bucket 3

Calculate inter-frames cosine distance For similar person distance is less than equal 15 degree

Md Abdullah Al Hafiz Khan (UMBC) SensePresence June 15, 2015 18 / 27

Case 2: People are not Conversing

Change point to capture the locomotive movements Use change points to find stray movements Baysian changepoint detection algorithm

◮ Calculate a-priori probability of two succesive change points at

distance d (run length)

◮ Gaussian based log-likelihood model to compute log-likelihood of

the data sequence [s,d] where no change point has been detected.

◮ Calculate log-likelihood for the entire signal S[t,n], log-likelihood of

data sequence Ss[t, s] where no change point occurs, π[i, t] log-likelihood where change point occurs

◮ summing up log-likelihoods for that sequence at time t ◮ set threshold δth ◮ Count number of change points to assign movement score Md Abdullah Al Hafiz Khan (UMBC) SensePresence June 15, 2015 19 / 27

Change Point Detection Result

5 10 15 20 25 30 500 1000 1500 2000

Magnitude No of Samples

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 500 1000 1500 2000

Probabilities No of Samples

Change point with probability values Count the number of changepoint as movement score Set threshold probability to eliminate few changepoint

Md Abdullah Al Hafiz Khan (UMBC) SensePresence June 15, 2015 20 / 27

Experimental Setup and Results

Data Collection: Natural conversation data collected and make it properly anonymized lab meeting, general discussion in lobby/corridor Data collection was 1-10 persons (with 5 males and 5 females) with age group of 18-50 years Audio sampling rate 16kHz at 16 bit PCM Locomotive sampling rate 5kHz Evaluation Metric: We use the average error count as the normalized predicted

• ccupancy metric

Error Count: |EC−AC|

N

where EC, AC, N denote the estimated people count, actual people count and number of samples respectively We use absolute value in order to avoid any positive or negative contribution

Md Abdullah Al Hafiz Khan (UMBC) SensePresence June 15, 2015 21 / 27

Occupancy Counting Results

0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2

10 15 20 25 30

Average Error Count Similarity Measure Threshold (degree)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 2 3 4

Average Error Count Distance Number of Speakers Table Pocket

Left figure depicts the effect of cosine distant similarity measures Similarity measure threshold is 15 degree Right figure reports the average error count distance 0.5 with respect to different phone positions

Md Abdullah Al Hafiz Khan (UMBC) SensePresence June 15, 2015 22 / 27

Occupancy Counting Results

0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 1 2 3

Average Error Count Distance Distance (m)

0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 3 4 5 6 7 8 10

Average Error Count Distance Number of Speakers

Left figure depicts that error count increases as leader’s distance from other occupants increases Right figure presents speaker counting performance (both

• verlapped and non-overlapped conversation)

Md Abdullah Al Hafiz Khan (UMBC) SensePresence June 15, 2015 23 / 27

Occupancy Counting Results

0.2 0.4 0.6 0.8 1 1.2 2 4 6 8 10 12

Binary Occupancy Sensor Number Prediction Ground Truth

2 4 6 8 10 12 2 4 6 8

Number of Occupants

Test Cases

• Acc. Estimated

Count

• Acc. Ground Truth

Acoustic Estimated Count Acoustic Ground Truth Combined Count Combined Ground Truth

Left figure shows binary occupancy counting Right figure presents locomotive augmented acoustic occupancy counting Example, 6 people converse and 4 remains silent. Acoustic sensing estimates 5 and locomotive sensing estimates 4. So total

• ccupancy 9 out of 10 people

Md Abdullah Al Hafiz Khan (UMBC) SensePresence June 15, 2015 24 / 27

Comparison with existing methodology

Number of Speakers Crowd++ (Error Count) Sense Presence (Error Count) 2 0.5 0.167 4 2.33 0.5 6 2.5 0.83 Average 1.78 0.5 Average error count distances for Crowd++ and SensePresence SensePresence accuracy increase more than 3 fold of Crowd++

Md Abdullah Al Hafiz Khan (UMBC) SensePresence June 15, 2015 25 / 27

Discussion & Future Work

Innovative system to infer number of people in a location. Unsupervised speaker count Posit changepoint detection algorithm to detect binary occupancy Context aware client-server based architecture Use smartphone’s microphone and accelerometer to count people Average error count 0.76 In future, we will Explore energy consumption Will try to add modality by adding location information Privacy issues can be resolved

Md Abdullah Al Hafiz Khan (UMBC) SensePresence June 15, 2015 26 / 27

Thank You

Md Abdullah Al Hafiz Khan Mobile Pervasive & Sensor Computing Lab University of Maryland Baltimore County email: mdkhan1@umbc.edu Website: http://userpages.umbc.edu/ mdkhan1/

Md Abdullah Al Hafiz Khan (UMBC) SensePresence June 15, 2015 27 / 27