Data Fusion in Sensor Networks Hugh Durrant-Whyte ARC Federation - - PowerPoint PPT Presentation

Slide 1 IPAM Sensor Networks Jan. 2007

Data Fusion in Sensor Networks

Hugh Durrant-Whyte ARC Federation Fellow, Research Director ARC Centre of Excellence for Autonomous Systems The University of Sydney, Australia hugh@cas.edu.au

ARC=Australian version of NSF

Slide 2 IPAM Sensor Networks Jan. 2007

Data Fusion in Autonomous Networks

• Decentralised Data Fusion (DDF)
• The DDF paradigm
• Decentralised Bayes and information fusion
• Fusion Challenges (ANSER II)
• Learning mixed sensor features
• Building general density models
• Control Challenges (RCS-18)
• Sensor and communication management
• Information in system design
• Future Challenges

Slide 3 IPAM Sensor Networks Jan. 2007

Decentralised Data Fusion (DDF)

• A set of Network Data Fusion

Methods:

• Ad Hoc Network
• Fusion at Sensor/Platform
• No Central Fusion Site
• Fully Scalable
• Decentralised Algorithms

Using the Information Filter:

• For Target Tracking
• For Cooperative Control
• For Cooperative Navigation

Target states (different bandwidths) Sensor Fusion Sensor Fusion Sensor Fusion To other network components Sensor Fusion

Slide 4 IPAM Sensor Networks Jan. 2007

Bayes and Information Fusion

Bayes:

( ) ( ) ( )

∏

−

=

j j k k

k k z P k P C k P ) ( | ) ( | ) ( . | ) (

1

x Z x Z x

Posterior Prior Sensor Likelihoods

• x(k) is the state at time k
• zj(k) is the jth sensor observation at time k
• Zk is the sequence of observations up to k

)) ( ( )) ( | ) ( ( k k k z P

j j

x x Λ =

Sensors contribute Likelihoods on x(k)

Slide 5 IPAM Sensor Networks Jan. 2007

Data Fusion: Distributed Sensing

...... ......

X

z1 zn zi Fusion Centre P(zi|x)

Sensor i

P(z1|x)

Sensor 1

P(zn|x)

Sensor n

Λ1(x) Λi(x) Λn(x)

P(x|Zn)=C P(x) Π Λi(x) n

i=1

P(x)

Slide 6 IPAM Sensor Networks Jan. 2007

Data Fusion: Distributed Fusion

......

X

z1 zn Fusion Centre P(z1|x)

Sensor 1

P(xk|z1) P1(xk|z) P(zn|x)

Sensor n

P(xk|zn) Pn(xk|z) P(xk|Zn) P(xk|Zn)=C P(xk-1) Π n

i=1

Pi(xk|z) P(xk-1|Zn) k k-1

Slide 7 IPAM Sensor Networks Jan. 2007

Data Fusion: Decentralised Fusion

......

X

z1 zn P(z1|x)

Sensor 1

P(xk|z1) P1(xk|z) P(zn|x)

Sensor n

P(xk|zn) Pn(xk|z) P(xk|Zn) Fusion Centre P(xk|Zn)=C P(xk-1) Π n

i=1

Pi(xk|z) P(xk-1|Zn) k k-1 Fusion Centre P(xk|Zn)=C P(xk-1) Π n

i=1

Pi(xk|z) P(xk-1|Zn) k k-1 P(xk|Zn)

Slide 8 IPAM Sensor Networks Jan. 2007

Bayes and Information Fusion

Bayes:

( ) ( ) ( )

∏

−

=

j j k k

k k z P k P C k P ) ( | ) ( | ) ( . | ) (

1

x Z x Z x

Log-Likelihood:

( ) ( ) ( )

K k k z P k P k P

j j k k

+ + =

∑

−

) ( | ) ( ln | ) ( ln | ) ( ln

1

x Z x Z x

Information, by construction, is additive

[ ]

) ( log ) ( x x P E H − =

Information: A measure of compactness:

⎥ ⎦ ⎤ ⎢ ⎣ ⎡ − = ) ( ) | ( log ) : ( x z x z x P P E I

Mutual Information: a priori measure

• f contribution to compaction:

Slide 9 IPAM Sensor Networks Jan. 2007

Bayes and Information Fusion

Bayes:

( ) ( ) ( )

∏

−

=

j j k k

k k z P k P C k P ) ( | ) ( | ) ( . | ) (

1

x Z x Z x

Log-Likelihood:

( ) ( ) ( )

K k k z P k P k P

j j k k

+ + =

∑

−

) ( | ) ( ln | ) ( ln | ) ( ln

1

x Z x Z x

Information, by construction, is additive Gaussian Case: (Fisher or Canonical) Information Form

) | ( ) | (

1

k k k k

−

= P Y

j j T j j k

H R H I

1

) (

−

=

) | ( ˆ ) | ( ) | ( ˆ

1

k k k k k k x P y

−

=

) ( ) (

1

k k

j j T j j

z R H i

−

=

Slide 10 IPAM Sensor Networks Jan. 2007

The Information Filter

Observation updates are simple sums (unlike KF):

∑

+ − =

j j k

k k k k ) ( ) 1 | ( ˆ ) | ( ˆ i y y

∑

+ − =

j j k

k k k k ) ( ) 1 | ( ) | ( I Y Y

[ ]

) ( ) | ( ) | ( ˆ ) | ( ˆ ) | 1 ( ˆ k k k k k k k k k Bu Y y Ω y y + + = +

) ( ) ( ) ( ) | ( ) | 1 ( k k k k k k k

T

Ω Σ Ω Y Y − = +

Time/Structure updates are Dual to state (KF) Observation Updates :

Slide 11 IPAM Sensor Networks Jan. 2007

DDF in Operation

• Network communicates Information
• Nodes fuse local observation and

communicated Information

• Channels communicate local

information gain (mutual information)

Sensor Fusion Sensor Fusion Sensor Fusion To other network components

Σ Σ

i(k) yi(k|k) yi(k|k-1) yi(k|k) ~

∆ypj(k|k)

Channel Filters Prediction Preprocess

∆ypi(k|k)

Channel Manager

Sensor Node

∆yqj(k|k)

Slide 12 IPAM Sensor Networks Jan. 2007

ANSERII: All-Source Bayesian Fusion

• Full Bayes DDF for fusion of heterogeneous data from UAVs,

UGVs, human and data base sources

• Model general feature types; trees, buildings, dams, etc
• Identify and label features, integrate human inferences
• Real-time exploitation of network data by air, ground, human

Slide 13 IPAM Sensor Networks Jan. 2007

Bayesian DDF Node Structure

zi(k)

Time Update (Convolution) Preprocess and Feature Extraction

Sensor Node P Q

Observation Update (Multiplication) Channel Filter Channel Filter Density Fitting Likelihood Model Channel Manager Assimilation (Multiplication)

Pi(z|x) Pi(z=zi(k)|x) P(xk|Zk-1,zi(k)) P(xk|Zk-1) P(xk|Zk-1,zi(k)) P(xk|Zk) P(xk|zQ,zP)

Slide 14 IPAM Sensor Networks Jan. 2007

Learning and Inference Process

Slide 15 IPAM Sensor Networks Jan. 2007

Step I: Model Generation

• NLDR on image patches
• Find low-D representation for

significant patches of interest

• Use VBEM to find number and

centre for mixture model

s z x

( ) ( ) ( ) ( )

s s x s x z s x z P P P P | , | , , =

Slide 16 IPAM Sensor Networks Jan. 2007

VBEM to Estimate Classifications

• Need to estimate number

and location of classes

• Variational Bayes

Procedures:

• Simultaneous

estimation of class number and centre

• Can deal with order 105

state dimensions

• Unsupervised/semi-

supervised classification

• f sensory data
• Automatic generation of

Likelihood Function

Slide 17 IPAM Sensor Networks Jan. 2007

Resulting Location Likelihood Models

ANSER 2

• Mixture Model for Location

Parameters and Likelihoods

Slide 18 IPAM Sensor Networks Jan. 2007

Resulting Class Likelihood Models

( ) ( ) ( )

s s x s x z P P P | , |

IPAM Sensor Networks Jan. 2007

Step II: Model Inference (Air)

P(z | x,s)=P(z|x=[x,s]) is the required likelihood for inference

Slide 20 IPAM Sensor Networks Jan. 2007

Step II: Model Inference (Ground)

P(z | x,s)=P(z|x=[x,s]) is the required likelihood for inference

Slide 21 IPAM Sensor Networks Jan. 2007

ANSER II Also Uses Data-Base and Human-derived Information

• Human operator input:
• Metric Information
• Labels, Context
• On-line estimation of

“operator likelihood”

• Hyperspectral data “node”

Slide 22 IPAM Sensor Networks Jan. 2007

Component-Based Middleware for Deployment (ORCA)

Slide 23 IPAM Sensor Networks Jan. 2007

Mission System Implementation

System Video

Slide 24 IPAM Sensor Networks Jan. 2007

ANSER II Common Operating Picture

Slide 25 IPAM Sensor Networks Jan. 2007

Fusion: Future Challenges

• Feature Modelling: Finding “x”
• Generalising single-sensor feature models
• Learning mixed-modality feature models
• Learning and refining abstractions (MI?)
• Probabilistic Fusion: Finding “P”
• High-dimensional density estimation
• Find a general and efficient density family,

closed under fusion operations

• Data association with general densities
• Systems design: Finding “z”
• Use information to do system design,

assembly and reconfiguration

Slide 26 IPAM Sensor Networks Jan. 2007

Mutual Information & Control

• Mutual Information or information gain, is

exactly what is communicated in the DDF

• Can be exploited in sensor management,

communications and platform control

Σ Σ

i(k) yi(k|k) yi(k|k-1) yi(k|k) ~

∆ypj(k|k)

Channel Filters Prediction Preprocess

∆ypi(k|k)

Channel Manager

Sensor Node

∆yqj(k|k)

Slide 27 IPAM Sensor Networks Jan. 2007

Example Cooperative Control

• Inherits DDF

properties:

• Scalability
• Survivability
• The trajectory that

maximises information

• Information shared (DDF)

Slide 28 IPAM Sensor Networks Jan. 2007

RCS-18: Future Cooperative UAVs

• How best to use tactical UAV

fleets ?

• A list of candidate targets of

interest

• Coordinate a UAV fleet with

mixed sensors to:

• Locate,
• identify and
• prosecute targets
• Demonstrate this

Slide 29 IPAM Sensor Networks Jan. 2007

Set-up

• DDF Enabled on all platforms
• Mutual information on target

location and IDs

• A set of UAV manoeuvres:
• Point-to-Point
• Orbits
• K-step look-ahead

e

Orbit s

R d V

ϕ ϕ

ω ϕ =

∫

I I

Slide 30 IPAM Sensor Networks Jan. 2007

Targets

Slide 31 IPAM Sensor Networks Jan. 2007

Two Vehicle Demonstration

Slide 32 IPAM Sensor Networks Jan. 2007

Extending the Paradigm

• Other information-maximising

controllers

• Resource use (platform,

communications)

• Target cuing, hand-off, etc
• Search, Exploration

Slide 33 IPAM Sensor Networks Jan. 2007

Control: Future Challenges

• Dealing with Constraints
• Process constraints
• Time constraints (rendezvous)
• Cooperative Planning
• Heterogeneous platforms:

Who does what and when

• Re-tasking

Slide 34 IPAM Sensor Networks Jan. 2007

Future Challenges

• Theory
• Spatial Scales
• Temporal Scales
• The autonomous sensor
• The application
• Mine picture compilation
• Large-scale sub-sea surveys

Slide 35 IPAM Sensor Networks Jan. 2007

Mine Picture Compilation

Multi-Spectral Geophysical Airborne Laser Face Radar

Slide 36 IPAM Sensor Networks Jan. 2007

Large-Scale Sub-Sea Surveys

Acoustic Visual Laser Chemical Temperature