Parallel Algorithms and Optimization for Multi-Aperture Image - - PowerPoint PPT Presentation

1 1

Parallel Algorithms and Optimization for Multi-Aperture Image Superresolution Reconstruction

Bob Plemmons

with T. Guy and P. Zhang Wake Forest and the Project Research Team

1

Sponsored by ARO/DTO/IARPA

Outline

 Introduction: Comments on Superresolution (SR)

• Fusing arrays of images

PERIODIC Research Project  Inverse Problems Considered

• Block Toeplitz Matrix Structures & Parallel SR

Algorithms  Application-specific integrated circuit platforms

• FPGA implementation

 Test Results

2

3

Image resolution describes the detail an image holds. The following figure provides how the same image appears at different pixel resolutions. Example without noise or blur: Superresolution (SR) – digital means for increasing spatial resolution

• Low resolution images should be different “looks” of the same scene.
• Our interest – HR image for arrays of LR images taken using a single detector

camera – snapshot image.

INTRODUCTION - Image Resolution

4

Multiple Aperture Imaging

• Limited utility in the analog age of cameras since

photochemistry provides no means for fusing images

• Insure against failure of a single aperture
• Increases the field of view (about 180º x 120º for us)
• Allows acquisition of multispectral, polarization, and

multiscale data

• Emerging computational systems lead to performance

advantages

• Multiple apertures common in living systems

(including us)

Compound Lens of Dragonfly: A Multiple Aperture System

History: TOMBO: Multi-aperture Camera

“Thin observation module by bound optics(TOMBO),”

• J. Tanida, et al., Applied Optics, 2001

(Practical Enhanced Resolution Integrated Optical Digital Imaging Cameras) PERIODIC Project

7

2005 – 2009

Our Compound Lens Camera for Biomedical Studies

• A multi-spectral 18 lens imager
• 18 narrow line-width filters (400-800 nm)
• Integrated illumination
• Single large format 10.6 Mpixel CCD
• Used at: WFU Institute for Regenerative

Medicine

Printing Skin for Grafting

9

3 x 3 Array Prototype Camera

3 x 3 LOW-DEF images post- processed into one

• r more HIGH-DEF

images

10

Deblurring

Superresolution Computations

• l = up-sampling factor
• Low resolution images should be different “looks” of the same scene.
• They should have different sub pixel shifts.

11

Image Registration is the process of spatially matching two images, i.e., the reference and target images, so that corresponding coordinate points in the two images correspond to the same physical region of the scene being imaged.

Image Registration

Followed by interpolation (splines, etc) onto high resolution grid.

INVERSE PROBLEM CONSIDERED

12

13

Superresolution/Deblurring – Forward Model

• f is a target high-resolution representation for object,
• Sj represents the translation of image f due to the

relative position of lenslet j with respect to the object and the detector (shift)

• Hj is a blurring operator associated with lenslet j,
• D is a decimation matrix,
• gj is the low-resolution image associated with lenslet j,
• ηj describes noise process associated with jth image.
• Compact notation: g = Af + η

Image Formation Model: A low-resolution image, gj formed by the jth lenslet is given by: gj = D Hj Sj f + ηj

14

Inverse Problem

• Produce a high-resolution image from low-resolution lenslet

images. g A m = l2 , where l is the up-sampling factor.

• The blurring, decimation, and shift matrices need to be
• determined. Various types of penalty (regularization) terms can be
• used. Speed is a important in our case (FPGA implementation).

) ( ) (

2 2 2 2 1 1 2 1

f R f S DH S DH S DH g g g f J

m m m

λ +             −             =  

)} ( { argmin ˆ f J f

f

=

15 15 15

Superresolution Reconstruction Example

Up-sampling factor 4. Sixteen LR images taken using PERIODIC camera. The sixteen 128 x 128 images used to reconstruct a 512 x 512 HR image. l = 4. Reconstructed HR LR Subimage

16 16 16 16

Computation Bottleneck of SR

 Concatenate the product matrices into matrix A.

 Inverse problem:

• Large matrix computation: given sixteen 128 x 128 LR images, we construct

a 512 x 512 HR image. Normally, matrix A would be of a size 262,144 x 262,144 . We seek a sparse solution method.

The original matrix A has a two-level tri-diagonal block Toeplitz structure, and can be reduced to 9 smaller matrices,

• f size l2 x l2, where l is the up-sampling factor.

We use nine 16 x 16 matrices for the example above. Key step is rearranging the arrays based on properties of and produce a structured matrix

2 2 2 2

) ( min f g Af f J λ + − =

, ,

i

H D

i

S . A

17 17 17

Structures of Superresolution Matrices

Without blurring, A = DS.

For the array imaging SR problem there exists permutation matrices P and Q such That has the two-level block Toeplitz structure:

AP QT

18 18 18

Structures of Superresolution Matrices

With blurring, A = DHS. PTATAP has the same structure as QTAP below, two-level block Toeplitz.

19 19 19 19

Designing Algorithms by Exploiting Matrix Structures

Without blurring, A = DS. Notation: l = l

After even-odd (red-black) permutations in both rows and columns, matrix ATAf = ATg becomes

Solve with Combination of Block Gauss-Siedel (BGS) and Cyclic Reduction (CR).

20 20 20

Designing Algorithms by Exploiting Matrix Structures

With blurring, A = DHS, with spread of H no more than 2l +1 pixels. Again, solve with Block Gauss-Siedel method.

21 21 21

Features of the Algorithms

 Parallelism: At each step, we solve n/2l much smaller scale sub- problems that can be solved independently by many processors. This greatly enhances the throughput of the system.  Small scale computations: The main operations involved are matrix-vector multiplications on the scale of l2.  FPGA-implementation: With these algorithms, the SR problem is now practical to solve on parallel systems like FPGAs.

22

Application-specific Hardware Implementation

Curt Petersen, EM Photonics Inc.

23 23

Array Based Digital Super Resolution Hardware Implementation – High is Best

Estimated Performance Affordability (1/cost) Power (FLOPS/watt) Flexibility

GPUs Very High Low

(**GPU development systems for embedded applications are not yet available)

Very Low Very High FPGAs High Medium High High DSP High Medium Medium High Multicore MPs Medium High Low Very High CELL High Medium Very Low Medium

24 24

Array Based Digital Superresolution Hardware Implementation

FPGA Development Platform Features

• Virtex 5 SX50T FPGA
• 52K Logic Cells
• 594KB BlockRAM
• 288 DSPs
• 480 IO
• 256MB DDR2 SDRAM
• DVI Output
• Gigabit Ethernet
• Serial Terminal Debug
• USB JTAG Programming

25

Digital Super Resolution using FPGA: System Architecture

System Control

(RISC Soft Processor)

Super Resolution Kernel Internal FPGA Memory External SDRAM Memory Input/Output Internal FPGA Memory Packaging:

• Protection from the elements

and ruggedization.

• Redesign for desired form factor.

Power:

• Battery integration for portable use.

26

Digital Superresolution using Embedded Hardware: Processing Steps

(1) Image Capture

• Raw 10.6 Mpixel image

(2) Image Segmentation and Registration

• Segment and register raw image

into 16 low resolution (128x128) subimages

• Registration at sub-pixel resolution

(3) Digital Super Resolution Reconstruction

• Reconstruct 512x512 high resolution image
• Executed by FPGA
• Primarily matrix-vector multiplications

27

TEST RESULTS

28 28 28

A 10.6 MPixel raw image from the PERIODIC camera. We select sixteen 128 x 128 subimages to be used during reconstruction.

Superresolution Reconstruction

29 29 29

A set of 16 LR images taken using the PERIODIC camera. The sixteen 128 x 128 images were used to reconstruct a 512 x 512 HR image. Speedup factor 12. Reconstructed HR LR Subimage

Sample Test Result

Resolution enhancement ~ 2.0X

note: after registration

• nly 10 non-redundant

subimages

30 30 30

A set of 16 LR images taken using the PERIODIC camera. The sixteen 128 x 128 images were used to reconstruct a 512 x 512 HR image. (Walter Peyton) Reconstructed HR LR Subimage

Sample Test Result

31 31 31

Reconstructed HR registration centered

• n object in front

LR Subimage

Sample Test Result

Reconstructed HR registration centered

• n object in back
• Subpixel registration is sensitive to object distance. Whole scene digital superresolution would require a

spatially varying registration algorithm.

• Since depth information is embedded in the multiple subimages, this approach can be exploited to develop

a sensitive and inexpensive imaging system that captures 3D information (depth) and performs digital superresolution simultaneously.

Summary

 Superresolution (SR) – compound lens systems

 PERIODIC Research Project  Inverse Problem for array imaging SR  Block Toeplitz Matrix Structures & Parallel SR Algorithms  Application-specific FPGA implementation

• Test Results – Speedup factor about 12 vs

implementation on 3.1Ghz 32 bit laptop

32

33