A method of personal positioning based on sensor data fusion of - - PowerPoint PPT Presentation

A method of personal positioning based on sensor data fusion of wearable camera and self-contained sensors.

Makasakatsu Kourogi and Takeshi Kurata

National Institute of Advanced Industrial Science and Technology (AIST)

*1 *1 *2 *1 *2 HIT Lab., University of Washington

Purpose

Personal positioning for wearable PC user

Without external source of information such

as IR beacon or ultrasonic receivers.

Light-weight equipments without using

expensive highly specialized devices such as Inertial Measurement Unit (IMU).

Requirements

What are required for personal positioning?

Direction of the user’s head (camera)

A wearable camera is attached to the head.

Direction of the user’s body

The direction to which the user moves.

Position of the user

Proposed method

Source of information

A wearable camera A 3-DOF attitude sensor (head) A 3-DOF attitude sensor (torso) Walking distance measurement unit

Walking behavior is recognized and measured. A kind of a smart pedometer.

Wearable camera Attitude sensors (head tracker)

Heading direction Gravitational direction

Dead-reckoning module based on walking

Accelerometers (3-axis) Gyro-sensors (3-axis) Magnetometers (3-axis) Inclinometers (3-axis)

Proposed method: head direction

A head tracker (InterTrax2) is used to acquire the direction of user’s head.

Incremental error is accumulated. Direction initialization is required.

A problem to be solved

Proposed method:

Position and head direction

Image taken from the camera is used to acquire the position and head direction.

Image is matched with pre-stored image

database of which position and direction of capturing is precisely known.

Wearable camera Image registration Image database Position, direction initialized “Not matched”

No initialization occurs

Proposed method: body direction

A self-contained sensors are used to acquire the body direction.

Gyro-sensors (3-axis)

Incrementally accumulate errors.

Magnetometers (3-axis)

Enable us to get absolute reference.

Accelerometers (3-axis)

Used to detect and measure walking behavior.

These sensors are attached to the user’s torso.

Problems to be solved

Earth’s magnetic field (EMF)

The magnetic field is easily disturbed by

electronic devices and appliances.

Thus, it is difficult to use magnetometers to

reliably acquire the absolute direction.

Proposed method: EMF

How to estimate reliability of the magnetic field?

Dip angle should be constant at given longitude and

latitude.

Earth’s magnetic field (EMF) Gravitational direction Horizontal direction Dip

Proposed method: EMF

How to estimate reliability of the magnetic field?

We take a strategy to see if the EMF’s dip angle is

continuously correct while in motion to test the reliability of the EMF.

EMF can be reliable if its dip angle is correct while the user is walking.

Walking behavior can be detected by accelerometers.

Proposed method: EMF

Sequence of dip angle while in walking

Computed from actual output of magnetometers 10 20 30 40 50 60 70 5000 10000 15000 20000 Time [msec] Angle [deg]

Data range when the magnetic field is reliable

True dip angle

Proposed method: EMF

What if EMF seems unreliable?

Gyro-sensors are used to compute the

direction by accumulation of attitude.

Proposed method:

relative position

Walking detection

A typical pattern of forward and vertical

acceleration while in walking behavior.

• 0.5
• 0.4
• 0.3
• 0.2
• 0.1

0.1 0.2 0.3 0.4 0.5 0.6 10000 10500 11000 11500 12000 12500 13000 13500 14000 Time [msec] Acceleration [G] Vertical acceleration Forward acceleration

Actual output of accelerometers

Proposed method:

relative position

Dead-reckoning is used.

Position movement is decided with relative

position with dead-reckoning of combination of pedometer and the body direction.

Proposed method: data integration

How to integrate sensory data?

Kalman filter framework is used.

Sensor data are fed into the framework and

statistically consistent results are acquired on each discrete time.

Proposed method

Kalman filter framework

Walking distance estimation module

Kalman filter update loop

Image registration module buffer

Output

Update

Gravitational vector Acceleration vector Magnetic vector State vector

DB

Angular velocity (torso) Update covariance

Test reliability

Measurement Input

Update gain Attitude angle (head) Image

Accumulation

Generate measurement Covariance matrix

• f measurement error

Covariance matrix

Proposed method

Kalman filter:

state vector and measurement vector

⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ ⎡ =

k k k k k k k k k

y x φ θ ψ γ β α x

Position Attitude (head) Attitude (torso)

⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ ⎡ =

k k k k k k k k k

y x φ θ ψ γ β α ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ y

Proposed method

Kalman filter: equations of state vector

k k k k k k k

u ⋅ + =

+

) (

| | | 1

x D x x ) ˆ (

1 | 1 | | − −

− + =

k k k k k k k k

x y K x x

1 1 | 1 |

) ˆ ( ˆ

− − −

Σ + Σ Σ =

vk k k k k k

K

Prediction: Update: Kalman gain:

Walking distance (as input vector) Covariance matrix of measurement noise

⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ ⎡ = sin cos ) (

| k k k k

ψ ψ x D

where

Proposed method

Kalman filter: equations of covariance matrix

1 | 1 | |

ˆ ˆ ˆ

− −

Σ − Σ = Σ

k k k k k k

K

wk k k k k

Σ + Σ = Σ +

| | 1

ˆ ˆ

Prediction: Update:

Covariance matrix of process noise

Experimental results:

Positioning

Starting position End position Position of resetting by image registration

Applications

Personal navigation Annotation overlay

Annotative information is overlaid with

input video frame to wearable display.

Implementation

System diagram

Web Image server Ethernet 3DM-G RS-232C InterTrax2 USB

MicroOptical ClipOn Display

VGA

Software Database client

Wireless LAN CCD camera

Wearable PC

IEEE802.11b

Database server

Access points

Server

Experimental results:

Positioning and annotative information

Demo video:

Conclusions:

EMF can be reliably used to estimate the direction. Positioning is achieved by the proposed method. The method can be used to overlay annotative information on input frame.