{ } = t x ( t ), X x ( t ) | 0 t t i i State - - PDF document

SLIDE 1

Hugh Durrant-Whyte 1 IPSN April 2005 Hugh Durrant-Whyte 2 IPSN April 2005

Data Fusion in Sensor Networks

Hugh Durrant-Whyte

ARC* Federation Fellow ARC* Centre of Excellence for Autonomous Systems The University of Sydney, Australia hugh@cas.edu.au *ARC=Australian Research Council

Hugh Durrant-Whyte 3 IPSN April 2005

Overview

• Decentralised Sensor Networks
• Data Fusion in Decentralised Networks
• Structure of the problem
• The essential DDF algorithm: SKIDS
• Communication and Model Distribution: ISSS, OxNav
• Timing and UAV demonstration: ANSER
• The general Bayesian DDF: ANSER II
• Management and Control in Sensor Networks
• Sensor Management
• Trajectory Control
• Search and Exploration
• Future Challenges

Hugh Durrant-Whyte 4 IPSN April 2005

Decentralised Sensor Networks

• Endogenous Systems:
• No central fusion
• No central comms.
• No global network

knowledge

• Probabilistic Algorithms
• Tracking
• Navigation
• Bayes estimation

Sensor Node Sensor Fusion Processor Communications Medium Hugh Durrant-Whyte 5 IPSN April 2005

Decentralised Data Fusion (DDF)

• State and state models
• Observations and likelihood models
• Distributed and decentralised estimation
• The DDF algorithm

Hugh Durrant-Whyte 6 IPSN April 2005

Data Fusion: State

• A track: of position and velocity for example
• A field property: temperature/pressure field for example
• A discrete property: Identity or label for example

{ }

t t t t

i i t

≤ ≤ = | ) ( ), ( x X x

A State is a shared across sensor nodes

SLIDE 2

Hugh Durrant-Whyte 7 IPSN April 2005

Data Fusion: Observations

• A track observation: range, bearing, velocity for example
• A local field observation: temperature/pressure for example
• A discrete observation: presence/absence for example

{ }

t t t t

i i t j j

≤ ≤ = | ) ( ), ( z Z z

An Observation is local to a sensor node

{ }

N j

t j t

∈ = | Z Z

Hugh Durrant-Whyte 8 IPSN April 2005

Data Fusion: The Sensor Model

( )

) ( | ) ( t t P

j

x z

( )

) ( ) ( | ) ( t x t t P

j

= x z

Model Generation

( )

) ( | ) ( ) ( t t z t P

j j

x z =

Model Inference

( )

x z |

j

P ) (t

j

z

( )

) (t

j x

Λ

Sensor Model couples local observations to common global state

Hugh Durrant-Whyte 9 IPSN April 2005

Data Fusion: The State Model

( )

) ( | ) (

1 − i i

t t P x x

State Model couples immediate states to global states Temporal Encoding

( )

1 1)

( | ) (

− −

=

i i i

x t t P x x

i-1 i Structural Encoding

( )

i i j

x P = x x |

j i j

Hugh Durrant-Whyte 10 IPSN April 2005

Data Fusion: The Essential Problem

( ) ( ) ( )

∏

= =

− j k j j k k k k

t z P t P C t P ) ( | | ) ( . | ) (

1

x z Z x Z x

Observation Update

( )

( ) (

)

) ( | ) ( ) ( | ) ( | ) (

1 1 1 1 1 − − − − −

∫

=

k k k k k k k

t d t P t t P t P x Z x x x Z x

Time/Structure Update Conditional Independence of Observations Require Marginal State at current time Distributed and Decentralised versions of these equations can be constructed with superficial ease

Hugh Durrant-Whyte 11 IPSN April 2005

Data Fusion: Distributed Sensing

...... ......

X

z1 zn zi Fusion Centre P(zi|x)

Sensor i

P(z1|x)

Sensor 1

P(zn|x)

Sensor n

Λ1(x) Λi(x) Λn(x)

P(x|Zn)=C P(x) Π Λi(x) n

i=1

P(x) Hugh Durrant-Whyte 12 IPSN April 2005

Data Fusion: Distributed Fusion

......

X

z1 zn Fusion Centre P(z1|x)

Sensor 1

P(xk|z1) P1(xk|z) P(zn|x)

Sensor n

P(xk|zn) Pn(xk|z) P(xk|Zn) P(xk|Zn)=C P(xk-1) Π n

i=1

Pi(xk|z) P(xk-1|Zn) k k-1

SLIDE 3

Hugh Durrant-Whyte 13 IPSN April 2005

Data Fusion: Decentralised Fusion

......

X

z1 zn P(z1|x)

Sensor 1

P(xk|z1) P1(xk|z) P(zn|x)

Sensor n

P(xk|zn) Pn(xk|z) P(xk|Zn) Fusion Centre P(xk|Zn)=C P(xk-1) Π n

i=1

Pi(xk|z) P(xk-1|Zn) k k-1 Fusion Centre P(xk|Zn)=C P(xk-1) Π n

i=1

Pi(xk|z) P(xk-1|Zn) k k-1 P(xk|Zn) Hugh Durrant-Whyte 14 IPSN April 2005

Decentralised Data Fusion: The Linear Gaussian Case

( )

Q w , ) ( N k →

) ( ) 1 ( ) ( k k k Gw Fx x + − =

( ):

) 1 ( | ) ( − k k P x x

( )

j j

N k R v , ) ( →

) ( ) ( ) ( k k k

j j j

v x H z + =

( ):

) ( | ) ( k k P

j

x z

( )

( )

) | ( ), | ( ˆ ); ( | ) ( q p q p p N p P

q

P x x Z x →

Hugh Durrant-Whyte 15 IPSN April 2005

The Information Filter

Bayes: Log-Likelihood: (Fisher or Canonical) Information Form:

( ) ( ) ( )

∏

−

=

j j k k

k k z P k P C k P ) ( | ) ( | ) ( . | ) (

1

x Z x Z x

( ) ( ) ( )

K k k z P k P k P

j j k k

+ + =

∑

−

) ( | ) ( ln | ) ( ln | ) ( ln

1

x Z x Z x

) | ( ) | (

1

k k k k

−

= P Y

j j T j j k

H R H I

1

) (

−

=

) | ( ˆ ) | ( ) | ( ˆ

1

k k k k k k x P y

−

= ) ( ) (

1

k k

j j T j j

z R H i

−

=

Hugh Durrant-Whyte 16 IPSN April 2005

The Information Filter

Observation updates are simple sums (unlike KF):

∑

+ − =

j j k

k k k k ) ( ) 1 | ( ˆ ) | ( ˆ i y y

∑

+ − =

j j k

k k k k ) ( ) 1 | ( ) | ( I Y Y

[ ]

) ( ) | ( ) | ( ˆ ) | ( ˆ ) | 1 ( ˆ k k k k k k k k k Bu Y y Ω y y + + = +

) ( ) ( ) ( ) | ( ) | 1 ( k k k k k k k

T

Ω Σ Ω Y Y − = +

Time/Structure updates are Dual to state (KF) Observation Updates :

Hugh Durrant-Whyte 17 IPSN April 2005

Implementation of the Decentralised Information Filter

Σ

{ in(k), In(k) } zn(k) k k-1 HTR-1

n n

Prediction yn(k|k-1), Yn(k|k-1)

^

yn(k|k), Yn(k|k) ~ ~

Sensor node n

Σ

~

[y1(k|k)-y1(k|k-1)]

Σ

{ i1(k), I1(k) } z1(k) HTR-1

1 1

Prediction

Sensor node 1

Σ

y1(k|k-1), Y1(k|k-1)

^

y1(k|k), Y1(k|k) ~ ~ k k-1

Σ

{ i2(k), I2(k) } z2(k) k k-1 HTR-1

2 2

y2(k|k), Y2(k|k) ~ ~ y2(k|k-1), Y2(k|k-1)

^

Prediction

Sensor node 2

Σ

~

[y2(k|k)-y2(k|k-1)]

~

[yn(k|k)-yn(k|k-1)] Hugh Durrant-Whyte 18 IPSN April 2005

SKIDS

• “European First Framework” Project 1986-

1991: Distributed/Decentralised Tracking in Civilian Environments

• Sensors
• Multiple cameras each with embedded

processing

• Optical barriers, acoustic sensing
• Algorithms
• DDF position/velocity tracking
• Distributed data association and identification
• Fully connected

SLIDE 4

Hugh Durrant-Whyte 19 IPSN April 2005

SKIDS (1989 Demo)

Hugh Durrant-Whyte 20 IPSN April 2005

Three Issues Arising From SKIDS

• Communication:
• Non-fully connected networks
• Time-varying and ad-hoc networks
• Communications management
• Modelling States:
• Partial and distributed models at nodes
• Estimation of field-like properties
• Non-Gaussian posteriors
• Control:
• Sensor Placement
• Sensor Management

Hugh Durrant-Whyte 21 IPSN April 2005

Communication: Operation of Sensor Nodes

• Nodes fuse information from:
• Local observations, Local predictions, and
• Communicated information
• Focus on Channels (the Channel Filter):
• Communicate local information gain
• Assimilate information gains from neighbourhood

Σ Σ

i(k) yi(k|k) yi(k|k-1) yi(k|k) ~ ∆ypj(k|k)

Channel Filters Prediction Preprocess

∆ypi(k|k)

Channel Manager

Sensor Node ∆yqj(k|k)

Hugh Durrant-Whyte 22 IPSN April 2005

Operation of Channels

( | )

i

I X Z

i

Z

( | ) ( | )

i i j

I X Z I X Z Z → ∩

( | )

i

I X Z

Hugh Durrant-Whyte 23 IPSN April 2005

( | )

i

I X Z Operation of Channels

( | )

i

I X Z

i

Z

( | )

i j

I X Z Z ∩

j

Z

( | )

i j

I X Z Z ∪

Hugh Durrant-Whyte 24 IPSN April 2005

Operation of Channels

i

Z

j

Z

( | )

i j

I X Z Z ∪

( | ) ( | )

i j i j

I X Z Z I X Z Z ∪ → ∩ ( | ) ( | )

i j i j

I X Z Z I X Z Z ∪ − ∩

( | )

i j

I X Z Z ∪

SLIDE 5

Hugh Durrant-Whyte 25 IPSN April 2005

Operation of Channels

( | )

i j

I X Z Z ∪

( | )

i j

I X Z Z ∩

( | )

i j

I X Z Z ∪

Hugh Durrant-Whyte 26 IPSN April 2005

Scaling The Network

( | )

i j

I X Z Z ∪ ( | )

i j

I X Z Z ∩ ( | )

i j

I X Z Z ∪ ( | )

i j

I X Z Z ∪ ( | )

Q

I X Z ( | )

Q

I X Z ( | )

i j Q

I X Z Z Z ∪ ∪

Hugh Durrant-Whyte 27 IPSN April 2005

Building Large Networks

• Information propagates through network
• Without increase in local bandwidth req.
• Within locality constraints of algorithms
• Broadcast, and tree networks:
• Optimal (central equivalent) results
• At the cost of network robustness
• Arbitrary and ad-hoc Networks:
• Common Information not captured by Channels
• Either local spanning trees or
• Conservative update policies (CI, Mutual Info.)

Hugh Durrant-Whyte 28 IPSN April 2005

Sub-Model Distribution

) ( ) ( k k

j j

x H z = ) ( ) ( k k

j j j

x C z =

j j j

T C H = ) ( ) ( k k

j j

x T x =

Identification of Locally Observable Sub-States:

+

=

i i i

k k T F T F ) ( ) (

+

=

i i i

k k T G T G ) ( ) (

Construction of Local Transition and Noise Models:

+

= =

j i j i j j i i j

k k T T V x V x ), ( ) (

Generation of Inter-Nodal Communication Models:

Hugh Durrant-Whyte 29 IPSN April 2005

ISSS: 1989-1993

• Objectives:
• Demonstrate scalability of DDF methods to systems

~100s sensors/nodes

• Explore network connectivity issues and communication

policies

• Implement and validate model decentralisation principles
• Fault detection and robust recovery
• Implementation
• A mock-up (nuclear) power plant model
• Primary water coolant, secondary air coolant
• Boiler, heat exchangers, pumps, by-pass circuits
• Measurement of pressures, temperatures, flows, etc

Hugh Durrant-Whyte 30 IPSN April 2005

ISSS: Sensor Network

• Network
• 250 sensors (temperature,

pressure, flow)

• 45 control points (valves,

power)

• 22 distributed processing nodes

SLIDE 6

Hugh Durrant-Whyte 31 IPSN April 2005

ISSS: Sensor Nodes

Hugh Durrant-Whyte 32 IPSN April 2005

ISSS: Results and Conclusions

• Communications
• Spanning tree algorithms work well in this slowly varying problem
• Optimal network communication provably not possible in general
• Model Distribution
• Models vary with time and must be re-learnt periodically
• Mutual Information provides the best method for doing this

Hugh Durrant-Whyte 33 IPSN April 2005

OxNav (1991-1995)

• Objectives:
• Modular, Decentralised

Mobile Robotics

• Fully Modular Sensors

and Actuators

• Fully Decentralised

Navigation and Control

Drive System Nodes Sonar Sensing Nodes

Hugh Durrant-Whyte 34 IPSN April 2005

Decentralised Navigation

• A Feature-Based

Navigator

• Using a tracking sonar

to lock-on and track stable features

• Decentralised

Information filter for estimating platform and feature locations (origin

• f the SLAM algorithm)
• Bayes estimator for

feature identity/type

Hugh Durrant-Whyte 35 IPSN April 2005

Decentralised and Scalable Control

• Modular, self-contained driven

and/or steered wheel units

• A reference with respect to a

“virtual” wheel

• No local knowledge of other

wheels or platform geometry

• Decentralised information filter

closes loop

• A reconfigurable decentralised

PID/LQG controller

Hugh Durrant-Whyte 36 IPSN April 2005

OxNav: Results and Conclusions

• Decentralised Estimation
• Fundamental Bayes form gives useful insight
• Structure of hybrid mobile/stationary tracking prob.
• Decentralised Control
• Mutual information is key to sensor management

and active data association

• Bayes, rather than complementary LQG is the right

approach to general control problems

• In 1995, limited interest in sensor networks so

I emigrated to Australia to do Field Robotics ☺

SLIDE 7

Hugh Durrant-Whyte 37 IPSN April 2005

ANSER (1999-2003)

Autonomous Navigation and Sensing Experimental Research

• Objectives:
• To deploy a fully decentralised data fusion system on a group of four or

more UAVs

• To demonstrate functions of target tracking and simultaneous localisation

and mapping, decentralised on many sensors in a network of platforms

• To demonstrate, algorithmically and practically, key network-centric

features: Modularity, Scalability, Flexibility and Survivability

Hugh Durrant-Whyte 38 IPSN April 2005

Flight Platforms

• Four Platforms – Delta Wing Configuration
• Max Speed – 80kts
• Payload Capacity – 20kg
• Wing Span – 3m
• Multiple Sensors per platform
• All modular pay-loads
• All parts interchangeable

Hugh Durrant-Whyte 39 IPSN April 2005

On-Board Components

Vehicle Bus (CAN) Flight Sensors GPS/IMU GPS/IMU and Flight Controller Flight Mode Switch Actuators Air-Ground Modem Environment Bus (CAN) Radar Node Air-to-Air Communications Radar Laser Node Vision Node Camera Laser Scanner

Flight Sensors Mission Sensors

Hugh Durrant-Whyte 40 IPSN April 2005 Hugh Durrant-Whyte 41 IPSN April 2005

Mission Planning System

Hugh Durrant-Whyte 42 IPSN April 2005

Overall DDF Communications Schematic

S1 S2 S3 S4 S1 S2 S3 S4 S1 S2 S3 S4 S1 S2 S3 S4

DDF DDF DDF DDF DDF DDF DDF DDF DDF DDF DDF DDF DDF DDF DDF DDF

Ground Station UAV1 UAV2 UAV3 UAV4

SLIDE 8

Hugh Durrant-Whyte 43 IPSN April 2005

Mission Control

Hugh Durrant-Whyte 44 IPSN April 2005

Multi-Vehicle Flights (2000-2001)

Hugh Durrant-Whyte 45 IPSN April 2005

3 Vehicle Flight Test (2002)

Hugh Durrant-Whyte 46 IPSN April 2005

ANSER Mission Data

Hugh Durrant-Whyte 47 IPSN April 2005

3 Vehicle – 4 Nodes Flight Test

Hugh Durrant-Whyte 48 IPSN April 2005

ANSER: Conclusions

• Information Communications is key:
• Timing, delay, asequent and burst communication
• Maintaining integrity, extensible network operation
• Channel and information management
• Data Fusion issues:
• Registration and platform bias estimation
• Cross-platform data association
• Weak target information not captured well by

information filter alone

• Commercial issues
• BAE Systems Chairman’s Gold Award
• Output integrated in to a number of on-going UK,

US and Australian Defence programmes

• (After 15 years of work !)

SLIDE 9

Hugh Durrant-Whyte 49 IPSN April 2005

ANSER II: 2004-2006

• Objectives
• General Bayesian DDF
• Heterogeneous

Information Sources

• Inferences from weak

sources

• Rapid exploitation of

network data

Hugh Durrant-Whyte 50 IPSN April 2005

Bayesian DDF Node Structure

zi(k)

Time Update (Convolution) Preprocess and Feature Extraction

Sensor Node P Q

Observation Update (Multiplication) Channel Filter Channel Filter Density Fitting Likelihood Model Channel Manager Assimilation (Multiplication)

Pi(z|x) Pi(z=zi(k)|x) P(xk|Zk-1,zi(k)) P(xk|Zk-1) P(xk|Zk-1,zi(k)) P(xk|Zk) P(xk|zQ,zP)

Hugh Durrant-Whyte 51 IPSN April 2005

Generation of Likelihood Models

s z x

( ) ( ) ( ) ( )

s s x s x z s x z P P P P | , | , , =

P(z | x,s)=P(z|x=[x,s]) is the required likelihood for inference

Hugh Durrant-Whyte 52 IPSN April 2005

Cross-Platform Fusion

• Bayesian DDF implemented

using GMMs

• Variational EM does local

estimation of GMM

Hugh Durrant-Whyte 53 IPSN April 2005

Human DDF Node

• Human operator input:
• Metric Information
• Labels
• Context
• On-line estimation of

“operator likelihood”

s

r

z x l

h

z

Hugh Durrant-Whyte 54 IPSN April 2005

Decentralised Sensor Management

• How to best acquire target information:
• Which sensor to point
• What direction to point the sensor in
• What trajectory for the aircraft
• What to communicate between platforms
• Require solutions that:
• Coordinate decentralised sensor actions
• Scale to large inhomogeneous networks
• Algorithms which are “information-seeking”

SLIDE 10

Hugh Durrant-Whyte 55 IPSN April 2005

Mutual Information Gain as a Control Metric

• Mutual Information is an a priori measure of average

Information gain following observation               = )) ( P( )) ( ) ( P( log E )) ( : ) ( ( t t t t t I x z | x z x

Measures “compression”

• f posterior density

Choose the sequence of observations z(t) which maximise

mutual information gain over a horizon

Observations depend on platform state x(t) State is governed by some control input u(t) Choose u(t) to maximise information gain

Hugh Durrant-Whyte 56 IPSN April 2005

Mutual Information in DDF

• Mutual Information or information gain, is exactly

what is communicated in the DDF

• Suggests a Decentralised (Greedy) management

and control algorithm:

• Order strategies according to local information gain
• Communicate options as information gain
• Select action which maximises gain for the group
• Appropriate to both sensor management and

platform control tasks

Hugh Durrant-Whyte 57 IPSN April 2005

Trajectory Control: A Canonical Example

• Bearings-only observation of a point-target

from a moving platform

      − − = ) ( ˆ cos ) ( ˆ cos ) ( ˆ sin ) ( ˆ cos ) ( ˆ sin ) ( ˆ sin ) ( ˆ 1 ) (

2 2 2 2

t t t t t t t r t θ θ θ θ θ θ σ β

β

I

For a single platform solutions are Helical

Hugh Durrant-Whyte 58 IPSN April 2005

Multiple Platform Management

• For a group of platforms:
• Solutions are parabolas
• Different initial locations

and prior information give different solution sets

Hugh Durrant-Whyte 59 IPSN April 2005

Two Vehicle Cooperative Tracking

Hugh Durrant-Whyte 60 IPSN April 2005

Decentralised Multi-Target Multi- Platform Tracking: Robustness and Scaling

SLIDE 11

Hugh Durrant-Whyte 61 IPSN April 2005

Control as Gradient Ascent in Information Space: The Surfing Analogy

Hugh Durrant-Whyte 62 IPSN April 2005

DDF/D-Control Algorithm Implementation

Conventional DDF Node Internode Negotiation Local Controller

Hugh Durrant-Whyte 63 IPSN April 2005

Cooperative Control for Target Detection

BAE Systems & UK MOD

Hugh Durrant-Whyte 64 IPSN April 2005

Nature of Information Maximisation Control

• Information gain problems:
• Are integral pay-off problems
• And turn out to have simple product/sum

structures that are easily distributed

• Information gain problems include:
• Target tracking/surveillance/reconnaissance
• Exploration and area covering
• Information gain admits useful solutions and

potential insights into the multi-agent coordination problem

Hugh Durrant-Whyte 65 IPSN April 2005

Exploration and Search

• Exploration/Search are information seeking activities:
• Find unseen areas
• Seek “interesting” things
• An integral pay-off problem with a DDF-like solution
• Heterogenous information sources
• Heterogeneous control

Hugh Durrant-Whyte 66 IPSN April 2005

Exploration: Multiple-Objectives

Information Localisation Time

SLIDE 12

Hugh Durrant-Whyte 67 IPSN April 2005 Hugh Durrant-Whyte 68 IPSN April 2005

Search: Multiple Sensors, Bayesian Models

• Objective: Minimise expected time to first

detection:

• Establish a prior for the area to be searched
• Communicate expected information gain for

different headings

• Select a path which minimises posterior entropy
• Motion models for priors can be used to exploit

domain knowledge

Hugh Durrant-Whyte 69 IPSN April 2005

Node-to-Node Negotiation

Hugh Durrant-Whyte 70 IPSN April 2005

Multi-UAV Search

• Cooperative Exploration,

Search and Tracking

• Scalable solutions
• Use of domain knowledge

AOARD/AFOSR

Hugh Durrant-Whyte 71 IPSN April 2005

Exploiting Domain Knowledge: Example of Wind-Gust Data

Hugh Durrant-Whyte 72 IPSN April 2005

Exploiting Domain Knowledge: Example of Soft Constraints

SLIDE 13

Hugh Durrant-Whyte 73 IPSN April 2005

Exploiting Domain Knowledge: Example of Hard Constraints

Hugh Durrant-Whyte 74 IPSN April 2005

Data Fusion in Sensor Networks: Where to Next ?

• There is scope for a general computational model of

data fusion in sensor networks:

• Bayesian/probabilistic
• Explicit information-theoretic communication
• Use of mutual-information to learn state coupling
• Deployable in modular and reusable form
• Future development of higher-level fusion functions:
• Bayesian network models for situation understanding
• Information-theoretic management and control
• Exploitation of human and other weak-source data
• Realistic and challenging application development
• Identification of key applications
• Sharing of data sets

Hugh Durrant-Whyte 75 IPSN April 2005