A Reusable Camera Interface for Rovers Daniel Clouse Jet - - PowerPoint PPT Presentation

A Reusable Camera Interface for Rovers

Daniel Clouse Jet Propulsion Laboratory California Institute of Technology Pasadena, California Workshop on Software Development and Integration into Robotics 14 April, 2007

Acknowledgements

• Coauthors - Issa Nesnas, Clayton Kunz
• Sponsor - NASA Mars Technology Program
• Ideas - Lorenzo Fluckeiger, Richard Madison,

Michael McHenry, Hans Utz

• Design Reviewers - I-Hsiang Shu, Jeffrey

Edlund, Nik Melchoir, Robert Steele

CLARAty Overview

Rover Acquire Image Goto Target 1 Swappable Algorithm or Robot Adaptation ATRV Jr. Rocky 7 ROAMS

Functional Layer Decision Layer

Declarative Activity Functional Abstraction Rocky 8 Explore Site Goto Target 3 Deploy Instrument Acquire & Analyze Navigator

Morphin

Locomotor

R8_Model

Motor

R8_Motor

Pose Estimator

SAPP

Pt Cloud IMU

ISIS

Target Tracker

Falcon

Camera

1394 Cam

Stereovision

JPLV

http://claraty.jpl.nasa.gov

Application Scenario

Show Movie of Single Cycle Instrument Placement

http://claraty.jpl.nasa.gov/man/overview/movies/movie_viewer.php?file=FY05/scip_05.mp4&w=720&h=480

Application Scenario

(Single Cycle Instrument Placement)

(a)

Target

(b)

time = t2 (avoiding an obstacle) time = t1

Designated Target Target Hand-

• ff

Camera Use Cases:

• Monocular multi-resolution nav

imaging for target tracking

• Synchronized binocular stereo from

navcams for target ranging and fine pointing

• Synchronized quad stereo imaging

from nav and hazcams for target hand-

• ff
• Synchronized binocular stereo from

hazcams for obstacle avoidance

History of Camera Interface

Revision 1

• Single threaded
• Did not allow for extension except along hardware axis

Revision 2

• Image acquisition is task safe

 But camera settings may be changed by another thread

• Bridge pattern to allow functional extensibility

 But numerous classes; hard to adapt; never extended

• Defined interface for setting/logging camera parameters

 But difficult to use, extend

Revision 2 Class Hierarchy

Camera Logical Camera Hierarchy Physical Camera Device Device_Group Camera_Group X_Camera_Group Generic Classes Adaptation Classes Hardware Classes X_Hw_Camera Camera_Impl Device_Impl Camera is extendible

Requirements

Common tasks are easy.

• Acquire an image
• Set/Get camera parameters

Full access to hardware. Avoid implementing unused features.

• Video

Synchronization within changing camera groups.

• To support Firewire limits

Task safety.

• For both acquire and parameter setting

A Bad Solution for Task Safety

Problems.

• User must reestablish param settings for every lock.
• Assumes user’s will write cooperative code.
• Deadlock / Starvation are possible.

Image<uint8_t> img1; X_Camera cam1(hw_cam1); cam1.lock(); cam1.set_brightness(0.35); cam1.acquire(img1); cam1.unlock();

Logical/Physical Cameras for Task Safety

Physical camera object represents a piece of hardware.

• Interface is specific to the camera hardware
• Parameter change happens immediately
• Acquire uses current hardware parameter settings

Logical camera object maintains a single user’s view.

• Base class defines a common interface for all logical cameras.
• Interface may be extended to support hardware-specific functions.
• Parameter values are cached
• Acquire atomically sets params in hardware and acquires image.

No special user code is required for task safety.

Image<uint8_t> img1, img2; X_Hw_Camera hw_cam1(id_unique_to_hw); X_Camera cam1(hw_cam1); cam1.set_brightness(0.35); cam1.acquire(img1); cam1.acquire(img2);

Revision 3 Class Hierarchy

Definition of Logical Camera Class

class Camera : public Device { public: virtual bool set_contrast(double gain_percent) = 0; virtual double get_contrast() const = 0; virtual bool set_brightness(double offset_percent) = 0; virtual double get_brightness() const = 0; virutal bool set_exposure(double seconds) = 0; virutal double get_exposure() const = 0; enum IMAGE_FORMAT { MONO8, YUV411, …, RGB8, MONO16, RGB16 }; virtual bool set_format(IMAGE_FORMAT format, int width, int height); virtual IMAGE_FORMAT get_format() const = 0; virtual int get_width() const = 0; virutal int get_height() const = 0; virtual void acquire(Image<uint8_t> & image, Time* timestamp = NULL, Feature_Map* feat_map = NULL) = 0; virtual void acquire(Image<uint16_t> & image, Time* timestamp = NULL, Feature_Map* feat_map = NULL) = 0; };

Synchronized Acquisition: Problem

Snap images from multiple cameras at the same instant. For framegrabber cameras, synch signal is on dedicated wire.

• The subset of synchable cameras is defined by the hardware.
• One framegrabber channel per camera, so bandwidth is not an issue.
• Configuration is generally static.

For Firewire cameras, synch signal is on the bus.

• Any subset of cameras on the same bus may be synced.
• Other constraints (bus bandwidth, DMA slots) limit the size of the sync set.
• Configuration is dynamic.

We need to support both models cleanly.

A Solution: Transient Camera Groups

Device_Group specifies a set of devices.

• Devices may be added/removed from the set dynamically.
• Device_Groups may be created/destroyed dynamically.
• Provides for controlling power to devices.

Derived Camera_Group specifies set of syncable logical cameras.

• A logical camera may belong to more than one Camera_Group.
• Acquire synced images by calling Camera_Group::acquire().

X_Camera_Group is derived for each camera implementation.

• Implements sync mechanism for particular hardware.
• Enforces hardware-specific constraints (DMA channels, bus bandwidth, which

framegrabbers are connected, etc).

Definitions of Grouping Classes

class Device_Group { public: unsigned int size() const; void append(Device& device); void remove(Device& device) throw(invalid argument); }; class Camera_Group : public Device_Group { public: virtual int acquire(vector<Image<uint8_t>* > & img_vec, vector<Time> * timestamp_vec = NULL, vector<Feature_Map*> * feature_map_vec = NULL); virtual int acquire(vector<Image<uint16_t>* > & img_vec, vector<Time> * timestamp_vec = NULL, vector<Feature_Map*> * feature_map_vec = NULL); };

Camera_Group Code Example

X_Hw_Camera hw_cam1(id1), hw_cam2(id2); X_Camera cam1(hw_cam1), cam2(hw_cam2); X_Camera_Group grp(cam1, cam2); cam1.set_brightness(0.35); cam2.set_brightness(0.35); Vector<Image<uint8_t> > images(2); grp.acquire(images);

Implementation of MR1394_Camera

Firewire cameras controlled via MindReady driver. Camera class hierarchy presents a simple interface to the user. The apparent simplicity hides some implementation problems:

• When possible, maintain cameras in video mode for fast

acquisition.

• Resource bookkeeping (DMA channels, bus bandwidth)

across multiple threads.

• Avoid changing camera parameters with each acquire.

Lessons Learned

Good interfaces don’t come easy.

• Thinking about the design is essential.
• But experience gained from implementation and use may lead to improvement.
• Don’t be afraid to re-implement.

Interface flexibility is not always a win.

• Example: Bridge pattern used in revision 2.
• Generality adds complexity, difficulty of maintenance.

Adding reusability as a requirement makes the implementation of an interface more complicated.

• Presentation of false simplicity to the user can make the user’s life easier.
• But makes the implementation more difficult.
• Increased implementation difficulty is only justifiable if offset by the savings

experience by large number of users, applications, etc.

Backup

Different Hardware Architecture

Actuator / Encoders / Potentiometers

Backplane (VME, PCI) Processor board Image acquisition boards Digital I/O board Analog I/O board Wireless ethernet

Video Switcher Gyroscopes

Accelero- meters Potentio- meters Synchronized stereo camera Monocular camera Synchronized stereo camera Monocular camera Analog signals Analog signals

Backplane or SBC Processor board or Single board computer (SBC) Serial bus / FireWire boards

FireWire Serial Bus / Digital signals Digital & Analog signals Serial Bus / Digital signals Microprocessors Digital I/O Analog I/O Serial comm Actuator / Encoders / Potentiometers Inertial Measurement Unit Science Instrument