Catching Events in Video Streams Mohan M. Trivedi Computer Vision - - PowerPoint PPT Presentation

Mohan M. Trivedi Computer Vision and Robotics Research Laboratory Electrical and Computer Engineering Department Jacobs School of Engineering Research Review February 28, 2003

Catching Events in Video Streams

Presentation Outline

Research Scope Intelligent Environments with Ubiquitous Vision Vision Systems for Intelligent Environments Event Catching in Intelligent Roads and Outdoor Spaces Intelligent Rooms and Indoor Spaces

Event Catching in Video Streams

Space Awareness (Static) Activity Awareness (Dynamic) Televiewing Summarization and Recall

Environmental Awareness

Intelligent Environments can:

• Develop and maintain awareness of events
• Adapt to the dynamic changes in their surroundings
• Interact in a natural, efficient and flexible manner with the users

Intelligent Environments

Video Streams Data or signals in spatio-temporal domain Events “Patterns” (semantically meaningful) in spatio-temporal domain Decisions Event based Actions

Event Catching in Video Streams

Research Highlights:

• Multiple Cameras
• Distributed Video Arrays
• Integrated Vision Systems
• Multiple level Abstractions
• Semantic Databases

KMET

Incident Detection and Management

Remote Agent: Outdoor Robotar for Tele-existence. Distributed architecture, multiple sensors, 2-way wireless multimedia streaming Main unit for on-scene incident verification (confirmed detection). Replaces current practice of waiting for CHP vehicle. Locally Active Little Agent: A team of small, flexible robots which work under the supervision of RA. Lower bit rate links, selected sensors, task-specific design. Team capability is very attractive KMET: Automatic discovery of available services. Advanced communication links, may also carry CMS (Changeable Message Sign) Roadside Active Network Adaptor: A controller for adaptive control for ramp metering and intersection control

DIVA: Distributed Interactive Video Arrays

DIVA: Traffic and Incident Management

UCSD Research Testbeds and Infrastructures

Command Node Base Node_1 Mobile Node_1 Wireless Test Zone HDR Base

I-5 Video/ Antenna Base

Televiewing Video

Research Testbeds and Infrastructures

–Multiple pan/tilt/zoom rectilinear cameras and Omnidirectional Cameras –Real-time 360° view of the area surrounding the pole –Wired to the lab using fibers –Sixteen high-bandwidth bidirectional Video steams accessible over internet –Televiewing for digital pan/tilt/zoom

Real-Time Shadow Segmentation

IEEE CVPR 2001; IEEE Trans. PAMI, 2003

( ) ( ) ( ) ( )2

min arg

i i i

x E x I x α α

α

− =

( ) ( ) ( ) ( )

i i i i

CD x E x x I x α − =

Motion Based Event Capture

Coronado Bridge San Diego Bay Interstate 5 UCSD Campus DIVA for Bridge

DIVA at Super Bowl

Riverbed-Qualcomm Stadium: Night Surveillance Party area Gas Lamp District: Crowd Monitoring Sea Port Command Center: Perimeter Sentry Night Surveillance Friars Road: Live Traffic Flow Notification

Perimeter Sentry

Real time histogram of occupancy of MZ is continuously updated. Log of the presence and of the movements inside the MZ is stored.

Semantic Queries: using Environmental Structure

Spatial Structure of A Highway Segment

Camera cluster

Monitor

In a camera cluster, when an incident occurs, the monitor can choose the “best” camera view and control its PTZ .. and even choose to “follow” the car responsible of the incident!!

Camera cluster

Monitor

In a camera cluster, the normal flow can be better handled by switching among cameras

Distributed Video Networks and Event Based Servoing

DIVA System Architecture

event-action tuple 1

EVENT DETECTION

E-A DATABASE event-action tuple 2 event-action tuple 3 event-action tuple N

....

ACTION DECISION MAKER DRIVING DIRECTIONS to robots ... FOCUS-OF-ATTENTION to secondary PTZ cameras ... INTERFACE from primary cameras ...

The DIAMOND architecture exhibits great

• flexibility. Using

the interface the user can create new event-action tuples

EVENT ACTION

stopped car in a given area (like emercency lane) Zoom in (Secondary camera 1) stopped car in a given area (like emercency lane) Zoom on the license plate (Secondary camera 2) incident detected Zoom on the incident (Secondary camera 1) incident detected (injuries) take a close of the injuries (Secondary camera 2) flat tire car Look if the driver needs help (Secondary camera 1) incident detected Take video from all the perspectives possible (Robot

• mnidirectional 1) – drive

toward the incident site

.... ....

Primary

Secondary

Primary

Secondary

Primary

Secondary

Primary

Secondary

“DIVA”--Event Driven Servoing

FI X E D Camera 1 FI X E D Camera 2 M OV I NG Camera 3

Camera Handover and Event Based Servoing

Simultaneous 3D tracking of multiple blobs Face orientation estimation

Kohsia Huang Capture of “interesting” events

Face recognition

Mohan Trivedi

time

whiteboard area Ivana Mikic Kohsia Huang Mohan Trivedi presentation question answer comment

Multiple Abstractions

MVA, 2003

Video Array for Ubiquitous Coverage

Rectilinear camera network Thermal Infrared

Original data Segmentation Voxel data

3D Voxel Reconstruction

Voxel labeling EKF prediction

Tracking

Initial estimate

Model initialization

Model refinement using the Bayesian network EKF update

Body Modeling and Movement Analysis System

( )

1

λ

H

d

( )

2

λ

S

d

. x y z

torso coordinate system world coordinate system 7 . 8 . = =

H S

d d

. . . . .

+ joint angle limits = valid postures Torso position: 3 parameters Torso orientation: 4 parameters Joint angles: 16 parameters = 23 parameters

Human Body Model

Key Features:

• 1. Completely automated system for motion

capture

• 2. Multi-resolution 3D voxel generation
• 3. Heuristic model initialization, refinement, and

tracking procedure

• 4. Robust voxel labeling procedure that handles

large frame to frame displacements

Potential Applications:

• 1. Advanced user interfaces
• 2. Video games and computer animation
• 3. Motion analysis for medical and sports

purposes

IEEE CVPR 2001; IJCV 2003

Body Modeling, Movement, Posture and Gait

Body and Movements

20 40 60 80 100 120 140

• 2

2 20 40 60 80 100 120 140

• 2

2 20 40 60 80 100 120 140

• 2

2 20 40 60 80 100 120 140

• 2

2 20 40 60 80 100 120 140

• 2

2 20 40 60 80 100 120 140

• 2

2 20 40 60 80 100 120 140

• 2

2 20 40 60 80 100 120 140

• 2

2 20 40 60 80 100 120 140

• 2

2 20 40 60 80 100 120 140

• 2

2 20 40 60 80 100 120 140

• 2

2 20 40 60 80 100 120 140

• 2

2 20 40 60 80 100 120 140

• 2

2 20 40 60 80 100 120 140

• 2

2

neck shoulder (L) shoulder (L) hip (L) hip (L) elbow (L) knee(L) neck shoulder (R) shoulder (R) hip (R) hip (R) elbow (R) knee(R)

Distributed Control Centers

Thanks !!

Website: cvrr.ucsd.edu

Event Catching in Video Streams