Sensor Fusion for Context Sensor Fusion for Context Understanding - - PowerPoint PPT Presentation

▶
sensor fusion for context sensor fusion for context
SMART_READER_LITE
LIVE PREVIEW

Sensor Fusion for Context Sensor Fusion for Context Understanding - - PowerPoint PPT Presentation

Jan 30, 2024 337 likes •569 views

Robotics Institute, Carnegie Mellon University Robotics Institute, Carnegie Mellon University Sensor Fusion for Context Sensor Fusion for Context Understanding Understanding Huadong Wu, Mel Siegel The Robotics Institute, Carnegie Mellon

slide-1
SLIDE 1

IMTC’2002, Anchorage, AK, USA 1 1 Robotics Institute, Carnegie Mellon University Robotics Institute, Carnegie Mellon University

IMTC-2002-1077 Sensor Fusion mws@cmu.edu

Sensor Fusion for Context Understanding Sensor Fusion for Context Understanding

Huadong Wu, Mel Siegel The Robotics Institute, Carnegie Mellon University Sevim Ablay Applications Research Lab, Motorola Labs

slide-2
SLIDE 2

IMTC’2002, Anchorage, AK, USA 2 2 Robotics Institute, Carnegie Mellon University Robotics Institute, Carnegie Mellon University

IMTC-2002-1077 Sensor Fusion mws@cmu.edu

sensor fusion

  • how to combine outputs of multiple

sensor perspectives on an observable?

  • modalities may be “complementary”,

“competitive”, or “cooperative”

  • technologies may demand registration
  • variety of historical approaches, e.g.:

– statistical (error and confidence measures) – voting schemes (need at least three) – Bayesian (probability inference) – neural network, fuzzy logic, etc

slide-3
SLIDE 3

IMTC’2002, Anchorage, AK, USA 3 3 Robotics Institute, Carnegie Mellon University Robotics Institute, Carnegie Mellon University

IMTC-2002-1077 Sensor Fusion mws@cmu.edu

context understanding

  • best algorithm for human-computer

interaction tasks depends on context

  • context can be difficult to discern
  • multiple sensors give complementary

(and sometime contradictory) clues

  • sensor fusion techniques needed
  • (but best algorithm for sensor fusion

tasks may depend on context!)

slide-4
SLIDE 4

IMTC’2002, Anchorage, AK, USA 4 4 Robotics Institute, Carnegie Mellon University Robotics Institute, Carnegie Mellon University

IMTC-2002-1077 Sensor Fusion mws@cmu.edu

agenda

  • a generalizable sensor fusion architecture

for “context-aware computing”

– or (my preference, but not the standard term) “context-aware human-computer interaction”

  • a realistic test to demonstrate usability

and performance enhancement

  • improved sensor fusion approach

(to be detailed in next paper)

slide-5
SLIDE 5

IMTC’2002, Anchorage, AK, USA 5 5 Robotics Institute, Carnegie Mellon University Robotics Institute, Carnegie Mellon University

IMTC-2002-1077 Sensor Fusion mws@cmu.edu

background

  • current context-sensing architectures

(e.g., Georgia Tech Context Toolkit) tightly couple sensors and contexts

  • difficult to substitute or add sensors, thus

difficult to extend scope of contexts

  • we describe a modular hierarchical

architecture to overcome these limitations

slide-6
SLIDE 6

IMTC’2002, Anchorage, AK, USA 6 6 Robotics Institute, Carnegie Mellon University Robotics Institute, Carnegie Mellon University

IMTC-2002-1077 Sensor Fusion mws@cmu.edu

Sensing hardware: cameras, microphones, etc. Environment situation: people in the meeting room, objects around a moving car, etc. humans understand context naturally & effortlessly

toward context understanding toward context understanding

Identification, representation, and understanding of context Adapt behavior to context traditional system Information Separation + Sensor Fusion

sensor sensor sensor

slide-7
SLIDE 7

IMTC’2002, Anchorage, AK, USA 7 7 Robotics Institute, Carnegie Mellon University Robotics Institute, Carnegie Mellon University

IMTC-2002-1077 Sensor Fusion mws@cmu.edu

methodology

  • top-down
  • adapt/extend Georgia Tech Context Toolkit

(Motorola helps support both groups)

  • create realistic context and sensor prototypes
  • implement a practical context architecture

for a plausible test application scenario

  • implement sensor fusion as a mapping of

sensor data into the context database

  • place heavy emphasis on real sensor device

characterization and (where needed) simulation

slide-8
SLIDE 8

IMTC’2002, Anchorage, AK, USA 8 8 Robotics Institute, Carnegie Mellon University Robotics Institute, Carnegie Mellon University

IMTC-2002-1077 Sensor Fusion mws@cmu.edu

context-sensing methodology: sensor data-to-context mapping context-sensing methodology: sensor data-to-context mapping

context

  • bservations

& hypotheses sensory output

                        ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ =            

m nm n n m m n

sensor sensor sensor f f f f f f f f f M L M O M M L L M

2 1 2 1 2 22 21 1 12 11 2 1

) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ? ? ?

slide-9
SLIDE 9

IMTC’2002, Anchorage, AK, USA 9 9 Robotics Institute, Carnegie Mellon University Robotics Institute, Carnegie Mellon University

IMTC-2002-1077 Sensor Fusion mws@cmu.edu

dynamic database

  • example: user identification and posture

for discerning focus-of-attention in a meeting

  • tables (next) list basic information about

environment (room) and parameters, e.g.,

– temperature, noise, lighting, available devices, number of people, segmentation of area, etc – initially many details are entered manually – eventually a fully “tagged” and instrumented environment can reasonably be anticipated

  • weakest link: maintaining currency
slide-10
SLIDE 10

IMTC’2002, Anchorage, AK, USA 10 10 Robotics Institute, Carnegie Mellon University Robotics Institute, Carnegie Mellon University

IMTC-2002-1077 Sensor Fusion mws@cmu.edu

space time social

  • utside

now activity schedule inside history

context

self, family, friends, colleague, acqauintance, etc.

mood agitation / tiredness stress concentration preferences physical information

merry, sad, satisfy… nervousness focus of attention habits, current name, address, height, weight, fitness, metablism, etc.

inside (personal information, feeling & thinking, emotional)

context classification and modeling context classification and modeling

location proximity time people audiovisual computing & connectivity

city, altidude, weather (cloudyness, rain/snow, temperature, humidity, barometer pressure, forecast), location and

  • rientation

(absolute, close to: building (name, structure, facilities, etc. knowledge), room, car, devices (function, states, etc.), …, vicinity temperature, humidity, day, date individuals or group (e.g. audience of a show, attendees in a cock-tail party): people interaction, casual chatting, formal meeting, eye contact, attention human talking (information collection), music, etc.; in- sight objects, surrounding scenery computing environment (processing, memory, I/O, etc., hardware/softw are resource & cost), network connectivity, communication bandwidth, communication change: travelling, speed, heading, change: walking/running speed, heading time of the day:

  • ffice hour,

lunch time, …, season of a year, etc. interruption source: imcoming calls, encounting, etc., … noise-level, brightness history, schedule, expectation social relationship

  • utside (environment)

sound processing: speaker recognition, speaking understanding image processing: face recognition,

  • bject

recognition, 3-D

  • bject

measureing location, altitude, speed,

  • rientation

ambient environment personal physical state: heart rate, respiration rate, blood pressure, blink rate, Galvanic Resistance, body temperature, sweat microphones cameras, infra- red sensors GPS, DGPS, serverIP, RFID, gyro, accelerometers, dead-reckoning network resource, thermometer, barometer, humidity sensor, photo- diode sensors, accelerometers, gas sensor biometric sensors: heart- rate/blood- pressure/GRS, temperature, respiration, etc., …

information: sensors

work body vision aural (listen/talk) hands

task ID drive, walk, sit, … read, watch TV, sight-seeing, …, people: eye contact content: work, entertainment, living chore, etc., … type, write, use mouse, etc., … interruptable…

activity

slide-11
SLIDE 11

IMTC’2002, Anchorage, AK, USA 11 11 Robotics Institute, Carnegie Mellon University Robotics Institute, Carnegie Mellon University

IMTC-2002-1077 Sensor Fusion mws@cmu.edu … …

Preference-table- user[Hd] Preference 144 lb (σ = 4 lb) Weight 5’6” (σ = 0.5” ) Height Huadong Wu (κ =1.0) Name

Preference-table-user[Hd] … …

Preference-table- user[Hd] Preference 144 lb (σ = 4 lb) Weight 5’6” (σ = 0.5” ) Height Huadong Wu (κ =1.0) Name

Background-table-user[Hd] … …

Preference-table- user[Hd] Preference NSH 4102 9:06AM-10:55AM Place Time Huadong Wu Name

History-table-user[Hd]

context information architecture: dynamic context information database context information architecture: dynamic context information database

6 (κ > 0.5) Detected people # User-table Detected users Current Device-table Devices 60 db (σ = 6 db) Noise level Brightness grade Light condition 72 ºF (σ = 3 ºF) Temperature Area-table Area

Room-table: NSH A417 User-table Detected user … …

4 (κ > 0.5) Detected people # Device-table Devices 60 db (σ = 6 db) Noise level Brightness grade Light condition 72 ºF (σ = 3 ºF) Temperature NSH A417 Of room

Inside Area

History-table- user[Chris] 10:45AM, 06/06/2001 Activity- table- user[Chris] [0.4, 0.9] Inside Background- table- user[Chris] Chris … Background- table- user[Alan] Background- table- user[Mel] Background- table-user[Hd] Background … History-table- user[Alan] History-table- user[Mel] History-table- user[Hd] history … 2:48PM, 06/06/2001 11:48AM, 06/06/2001 10:32AM, 06/06/2001 First detected … Activity- table- user[Alan] Activity- table- user[Mel] Activity- table-user[Hd] Activity … Inside Entrance Entrance Place … … [0.9, 0.98] Alan [0.3, 0.7] Mel [0.5, 0.9] Hd Confidence Name

User-table … … User-table Detected user

2 (κ > 0.5) Detected people # Device-table Devices 60 db (σ = 6 db) Noise level Brightness grade Light condition 72 ºF (σ = 3 ºF) Temperature NSH A417 Of room

Entrance Area

user PK user ID contact info. preference

  • conf. room usage

activity user activity PK user ID in meeting speeking head pose focus of attention

user PK user ID name contact info. preference

  • conf. room usage

activity conference room PK room ID function & location facilities usesage schedule current users brightness noise

slide-12
SLIDE 12

IMTC’2002, Anchorage, AK, USA 12 12 Robotics Institute, Carnegie Mellon University Robotics Institute, Carnegie Mellon University

IMTC-2002-1077 Sensor Fusion mws@cmu.edu

implementation

  • low-level sensor fusion done
  • sensing and knowledge integration via LAN
  • context maintained in a dynamic database
  • each significant entity (user, room, house, ...)

has its own dynamic context repository

  • dynamic repository maintains and serves

context and data to all applications

  • synchronization and cooperation among

disparate sensor modalities achieved by “sensor fusion mediators” (agents)

slide-13
SLIDE 13

IMTC’2002, Anchorage, AK, USA 13 13 Robotics Institute, Carnegie Mellon University Robotics Institute, Carnegie Mellon University

IMTC-2002-1077 Sensor Fusion mws@cmu.edu

system architecture to support sensor fusion for context-aware computing system architecture to support sensor fusion for context-aware computing

user context database appliance embedded OS database sever

gateway Internet Intranet

appliance embedded OS sensor smart sensor node sensor smart sensor node sensors applications sensor fusion sensor smart sensor node appliance embedded OS site context database context server higher-level sensor fusion applications lower-level sensor fusion sensors

slide-14
SLIDE 14

IMTC’2002, Anchorage, AK, USA 14 14 Robotics Institute, Carnegie Mellon University Robotics Institute, Carnegie Mellon University

IMTC-2002-1077 Sensor Fusion mws@cmu.edu

practical details ...

  • context type =>sensor fusion mediator
  • mediator integrates corresponding

sensors, e.g., by designating some primary others secondary based on

  • bserved or specified performance
  • Dempster-Shafer “theory of evidence”

implementation in accompanying paper

  • (white-icon components in following

cartoon inherited from Georgia Tech CT)

slide-15
SLIDE 15

IMTC’2002, Anchorage, AK, USA 15 15 Robotics Institute, Carnegie Mellon University Robotics Institute, Carnegie Mellon University

IMTC-2002-1077 Sensor Fusion mws@cmu.edu

system configuration diagram system configuration diagram

Dynamic Context Database

user - mobile computer site context database server site context server

Widget sensor Widget sensor Widget sensor Widget sensor SF mediator Aggregator SF mediator Aggregator context data Resource Registry context data Resource Registry Other AI algorithms Interpreter application application Other AI algorithms Interpreter AI algorithms Discoverer

Dempster-Shafer rule

slide-16
SLIDE 16

IMTC’2002, Anchorage, AK, USA 16 16 Robotics Institute, Carnegie Mellon University Robotics Institute, Carnegie Mellon University

IMTC-2002-1077 Sensor Fusion mws@cmu.edu

details inherited from GT TC

  • JAVA implementation
  • BaseObject class provides communication

functionality: sending, receiving, initiating, and responding via multithread server

  • context widgets, interpreters, and discovers subclass

from the BaseObject, and inherit its functionality

  • service is part of the widget object
  • aggregators subclass from widgets, inheriting their

functionality

slide-17
SLIDE 17

IMTC’2002, Anchorage, AK, USA 17 17 Robotics Institute, Carnegie Mellon University Robotics Institute, Carnegie Mellon University

IMTC-2002-1077 Sensor Fusion mws@cmu.edu

  • bject hierarchy and subclass

relationship in context toolkit

  • bject hierarchy and subclass

relationship in context toolkit

BaseObject Interpreter Discoverer Aggregator Widget Service

slide-18
SLIDE 18

IMTC’2002, Anchorage, AK, USA 18 18 Robotics Institute, Carnegie Mellon University Robotics Institute, Carnegie Mellon University

IMTC-2002-1077 Sensor Fusion mws@cmu.edu

focus-of-attention application

  • neural network estimates head poses
  • focus of attention estimate based on head

pose probability distribution analysis

  • audio reports speaker, assumed to be focus
  • f other participants’ attention
  • situation is not easy to analyze due to, e.g.,

dependence of behavior on discussion topic

  • initial results suggests we need more general

fusion approach than provided by Bayesian

slide-19
SLIDE 19

IMTC’2002, Anchorage, AK, USA 19 19 Robotics Institute, Carnegie Mellon University Robotics Institute, Carnegie Mellon University

IMTC-2002-1077 Sensor Fusion mws@cmu.edu

next paper ...

  • Dempster-Shafer approach ...
  • provides mechanism for handling

“belief” and “plausibility”

  • cautiously stated, generalizes Bayes’

Law a priori probabilities to distributions

  • (difficulty, of course, is that usually

neither the requisite probabilities nor the distributions are actually known)

slide-20
SLIDE 20

IMTC’2002, Anchorage, AK, USA 20 20 Robotics Institute, Carnegie Mellon University Robotics Institute, Carnegie Mellon University

IMTC-2002-1077 Sensor Fusion mws@cmu.edu

focus-of-attention estimation from video and audio sensors focus-of-attention estimation from video and audio sensors

slide-21
SLIDE 21

IMTC’2002, Anchorage, AK, USA 21 21 Robotics Institute, Carnegie Mellon University Robotics Institute, Carnegie Mellon University

IMTC-2002-1077 Sensor Fusion mws@cmu.edu

expectations expectations

context

AI rules Widget sensor Widget sensor e.g. Dempster-Shafer Belief Combination

Sensor fusion mediator

Observations & Hypotheses

Dynamic Configuration

Time interval: T Sensor list Updating flag … …

Expected Performance Boost Expected Performance Boost

  • 1. Uncertainty & ambiguity

representation to user applications

  • 2. Information consolidation &

conflict resolving for users

  • 3. Adaptive sensor fusion support

switch to suitable algorithms

  • 4. Robust to configuration change

— and for some to die gracefully

  • 5. Situational description support

— using more & complex context

slide-22
SLIDE 22

IMTC’2002, Anchorage, AK, USA 22 22 Robotics Institute, Carnegie Mellon University Robotics Institute, Carnegie Mellon University

IMTC-2002-1077 Sensor Fusion mws@cmu.edu

conclusions

  • preliminary experiments demonstrate

feasibility of context-sensing architecture and methodology

  • expect our further improvements via

– better uncertainty and ambiguity handling – fusion of overlapping sensors – context-adaptive widget capability – sensor fusion mediator coordinates resources – context information server supports applications