[PPT] - Methods to to Improve Improve Resolution Resolution of of PowerPoint Presentation

SLIDE 1

V Visual

isual I

Information

nformation P

Processing

rocessing Group Group

Methods Methods to to Improve Improve Resolution Resolution of

f

Compressed Compressed Video Video Sequences Sequences Based Based on

n

Observability Observability Maps Maps

Ioannina (Greece), October 2004

By L.D. Alvarez Ioannina, 10/04/2004

SLIDE 2

L.D. Alvarez

Ioannina, September 2004

Outline Outline

Introduction

A few things about myself. A few things about my city and university. A few things about my research group.

Brief introduction. Staff. Research areas.

Research

Introduction and notation. System model. Problem formulation. Optimization procedure. HR simultaneous estimation. Observability and predictibility. Results.

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

SLIDE 3

L.D. Alvarez

Ioannina, September 2004

INTRODUCTION INTRODUCTION

SLIDE 4

L.D. Alvarez

Ioannina, September 2004

Myself Myself

Personal information.

Luis David Alvarez Corral. Granada, Spain, 1977.

My background.

Computer Science Engineering degree at the University of

Granada in 2001.

Advanced Studies Diploma in 2003 (Master). Currently pursuing a Ph.D. degree at the department of

Computer Science and Artificial Intelligence of the University of

Granada. Granted by the Spanish Ministry of Education and

Science.

My research areas.

Enhancement of images and video. Superresolution problem from compressed observations using

the Bayesian framework (problem in compressed video within a Bayesian framework).

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

SLIDE 5

L.D. Alvarez

Ioannina, September 2004

The The city city and and the the University University

A brief history

Granada was iberian, roman, jewish, islamic (capital of the former

Nazari Kingdom) and Christian (last city on the Iberian Peninsula to be conquered from the Muslims in 1492 by the Catholic Kings).

The Christian conquest did not rob the city of its splendour as a

cultural centre, in which sciences and humanities found the best way to develop.

A historic University

The University of Granada was founded in 1531 by Emperor

Carlos V.

The University continued the tradition of the Arab University of

Yusuf I.

With 473 years of tradition, the University of Granada has been

an exceptional witness to history, as its influence in the city's social and cultural environment grew until it was to become, over a period of almost five centuries, an intellectual and cultural nucleus in Southern Spain in its own right.

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

SLIDE 6

SLIDE 7

SLIDE 8

SLIDE 9

L.D. Alvarez

Ioannina, September 2004

University numbers (year 2002)

Some 70,000 people are directly linked with the University of

Granada, among them students, teachers and administrative and service staff.

Number or degree courses 72 (in 24 university centres and 4

associated)

Degree and diploma students 60,000 Administration and Service Staff 1,578 Departments 107 Computers connected to Internet 9,500

For further information: http://www.ugr.es

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

SLIDE 10

L.D. Alvarez

Ioannina, September 2004

Research Research group group: VIP : VIP

The research group “Visual Information Processing”

(VIPG) consists of members of the Departament of Computer Science and Artificial Intelligence and the Department of Computer Languages at the University of Granada and the Departament of Informatics at the University of Jaén.

Staff:

Professor Nicolás Pérez de la Blanca. Professor Rafael Molina Soriano. 6 associate lecturers. 3 assistant lecturers. 1 associate investigator. 7 funded Ph. D. student.

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

SLIDE 11

L.D. Alvarez

Ioannina, September 2004

Some colaborations:

Institute of Astrophysics of Andalucía (IAA).

Prof. Brian Ripley (Oxford University, then at Strathclyde

University) on astronomical image restoration.

IMSOR (now IMM) at the Technical University of Denmark

n remote sensing classification problems.
Prof. A.K. Katsaggelos* (Northwestern University of

Evanston, Illinois).

Prof. F. Murtagh* (Queen’s University of Belfast).
Prof. E. P. Simoncelli* (New York University, New York).
Prof. Nikolaos Galatsanos* (Ioannina University, Greece).
Prof. E. Trucco (Herriot Watt University, Edinburgh) on 3D

modelization and videoconference.

Prof. Gustavo Marrero (IUMA, Las Palmas, Spain) on

hardware implementation of super-resolution algorithms.

* On areas such as image representation, image

restoration and reconstruction of (compressed/uncompressed) images and sequences.

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

SLIDE 12

L.D. Alvarez

Ioannina, September 2004

Some research areas:

Information retrieval from image databases. Feature

extraction from colour images.

Extraction of 3D information from video sequences for

multimedia applications.

Motion analysis. Applications to optical flow estimation and

motion segmentation.

Tracking algorithms and optical flow. Superresolution from compressed and uncompressed

stills and video sequences.

Visual-statistical approach to multi-scale multiorientation

representation models. Application to: texture modeling and synthesis, image restoration, image enhancement, model-based image coding.

Bayesian image restoration and reconstruction.

Applications to Astronomy and Medicine.

Fuzzy image processing. Image classification and pattern

recognition.

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

SLIDE 13

L.D. Alvarez

Ioannina, September 2004

For further information: http://decsai.ugr.es/vip

Members: http://decsai.ugr.es/vip/members Publications: http://decsai.ugr.es/vip/publications (where

you can get a copy of all our published work)

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

SLIDE 14

L.D. Alvarez

Ioannina, September 2004

RESEARCH RESEARCH

SLIDE 15

L.D. Alvarez

Ioannina, September 2004

Introduction Introduction and and notation notation

Objective: Our aim is to estimate high resolution video from a

compressed low resolution image sequence.

High-Resolution HR256x256 fk Low-resolution LR128x128 gk Compressed Low-resolution CLR128x128 yk

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

SLIDE 16

SLIDE 17

L.D. Alvarez

Ioannina, September 2004

System System model model

Objective: To estimate fl Relationship between one LR image and the corresponding

HR image:

H blurring A downsampling

gl

fl

yl

l l

AHf g =

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

l from 1 to the image sequence lenght (L)

SLIDE 18

L.D. Alvarez

Ioannina, September 2004

Key hypothesis: two frames will be similar if they are captured

at closely spaced time instances.

35

fk

35

fl

( )

1 3 · 2 , 1 3 · 2 3 · 2 3 · 2 , 1 3 · 2

35 · 1 35

x k l x k x k l x l

C ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎝ ⎛ + ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎝ ⎛ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎝ ⎛ = ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎝ ⎛ n f d f

( ) ( ) ( )

( )

y x n y x d y y x d x f y x f

k l y k l x k l k l

, , , , ,

, , ,

+ + + =

( )

k l k k l l , ,

C n f d f + =

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

SLIDE 19

L.D. Alvarez

Ioannina, September 2004

Then:

l l

AHf g =

( )

k l k k l l , ,

C n f d f + = ( )

k l k k l l , ,

C e f d AH g + =

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

SLIDE 20

L.D. Alvarez

Ioannina, September 2004

Relationship between one CLR image and the HR image we

want to estimate:

encoder decoder

( ) ( )

i i i k i i i k k DCT DCT k

C C T Q T y v y v AHf y

∑ ∑

∀ ∀ −

+ ⎥ ⎦ ⎤ ⎢ ⎣ ⎡ ⎟ ⎠ ⎞ ⎜ ⎝ ⎛ − =

, , 1

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

SLIDE 21

L.D. Alvarez

Ioannina, September 2004

( )

k l k k l l

C

, ,

n f d f + =

l l

AHf g =

HRl HRk LRl HRl LRl HRk

( )

k l k k l l

C

, ,

AHn f d AH g + =

( ) ( )

∑ ∑

∀ ∀ −

+ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎣ ⎡ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ⎥ ⎦ ⎤ ⎢ ⎣ ⎡ − =

i i i l i i i l l DCT DCT l

C C T Q T y v y v g y

, , 1

CLRl LRl CLRl HRk

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

( ) ( ) ( )

∑ ∑

∀ ∀ −

+ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎣ ⎡ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ⎥ ⎦ ⎤ ⎢ ⎣ ⎡ − + =

i i i l i i i l k l k k l DCT DCT l

C C C T Q T y v y v AHn f d AH y

, , , , 1

SLIDE 22

L.D. Alvarez

Ioannina, September 2004

We modelize part of the compression/decompression process

having into account that the dominant noise is the quantization noise and it follows a normal distribution of mean 0 and covariance matrix for the frame l-th, KQ,l [εQ,l ~ N(0,KQ,l)].

( )

l Q k k l l

C

, ,

ε f d AH y + =

CLRl HRk

( ) ( ) ( )

∑ ∑

∀ ∀ −

+ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎣ ⎡ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ⎥ ⎦ ⎤ ⎢ ⎣ ⎡ − + =

i i i l i i i l k l k k l DCT DCT l

C C C T Q T y v y v AHn f d AH y

, , , , 1

( ) ( ) ( ) ( )

l Q i i i l k k l i i i l k l k k l DCT DCT

C C C C T Q T

, , , , , , 1

ε y v f d AH y v AHn f d AH + − ≈ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎣ ⎡ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ⎥ ⎦ ⎤ ⎢ ⎣ ⎡ − +

∑ ∑

∀ ∀ − INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

SLIDE 23

L.D. Alvarez

Ioannina, September 2004

Best prediction for l-th CLR image: Assumption: motion compensation introduces some error.

∑

∀ i i i l

C y v ) (

,

( ) ( )

l MV k k l i i i l

C C

, , ,

η f d AH y v + =

∑

∀

( )

l MV i i i l l

C

, ,

τ + =∑

∀

y v y

( )

l Q k k l l

C

, ,

ε f d AH y + =

CLRl HRk

( )

l MV i i i l l

C

, ,

τ + =∑

∀

y v y

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

The noise ηMV,l is the sum of the error due to the quantization and the error due to the motion compensation. It follows a normal distribution of mean 0 and covariance matrix KMV,l [ηMV,l ~ N(0,KMV,l)].

SLIDE 24

L.D. Alvarez

Ioannina, September 2004

Problem Problem formulation formulation

HR algorithm based on:

Sub-pixel displacements. Aliasing preservation during compression.

We follow a maximum a posteriori estimation

approach (MAP) to recover the HR information using the CLR images. ( ) { }

V Y D f D f

D f

, | , max arg ˆ , ˆ

, k k

p

k

=

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

SLIDE 25

L.D. Alvarez

Ioannina, September 2004

Taking logarithms and assuming independency

between the image and the displacements, we have:

These functions incorporate into the framework:

information about the compression system, and, a priori knowledge of the original high-resolution image

sequence.

( ) { } ( ) ( ) ( ) ( ) [ ] ( ) [ ] ( ) [ ] { }

D f D f V Y V Y D f D f V Y V Y D f D f

D f D f D f

p p p p p p p

k k k k k k

k k k

log log , | , log max arg , , · , | , max arg , | , max arg ˆ , ˆ

, , ,

+ + = ⎭ ⎬ ⎫ ⎩ ⎨ ⎧ = =

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

SLIDE 26

L.D. Alvarez

Ioannina, September 2004

Frames are independent, and errors too: Modelization for each CLR frame from: Density function:

( ) ( )

∏

=

l k l k

p p D f y D f Y , | , |

( )

l Q k k l l

d C

, .

ε f AH y + =

( )

( ) ( ) ( ) ( )

⎭ ⎬ ⎫ ⎩ ⎨ ⎧ − − − ∝

− k k l l l Q T k k l l k l

C C p f d AH y K f d AH y D f y

, 1 , ,

2 1 exp , |

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

SLIDE 27

L.D. Alvarez

Ioannina, September 2004

Displacements errors are independent: Modelization for each CLR displacement: Density function:

( ) ( )

∏

=

l k l k

p p Y D f v Y D f V , , | , , |

( )

l MV k k l i i i l

C C

, . ,

η f d AH y v + =

∑

∀

( )

⎪ ⎭ ⎪ ⎬ ⎫ ⎪ ⎩ ⎪ ⎨ ⎧ ⎟ ⎠ ⎞ ⎜ ⎝ ⎛ − ⎟ ⎠ ⎞ ⎜ ⎝ ⎛ − − ∝

∑ ∑

− = − − = 1 . , 1 , 1 . ,

2 1 exp , , |

l i k k l i i l l MV T l i k k l i i l k l

C C C C p f d AH y v K f d AH y v Y D f v

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

SLIDE 28

L.D. Alvarez

Ioannina, September 2004

A priori information over the frames:

HR frames should be smooth (no compression artifacts). LR frames should be smooth (from smooth HR frames).

Density function:

2 1 k

f Q

2 2 k

AHf Q

( ) { }

2 2 2 2 1 1

exp ) (

k k k

p AHf Q f Q f λ λ + − ∝

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

SLIDE 29

L.D. Alvarez

Ioannina, September 2004

A priori information over the displacements:

HR displacements should be smooth.

Density function:

2 , 3 k l

d Q

( ) { }

∑−

∝

2 , 3 3 ,

exp ) (

k l k l

p d Q d λ

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

SLIDE 30

L.D. Alvarez

Ioannina, September 2004

Optimization Optimization procedure procedure

Expression to optimize: Optimization using a cyclic coordinate descent

procedure. ( ) ( ) ( ) ( )

[ ] {

( )

( )}

2 , 3 3 2 2 2 2 1 1 1 . , 1 , 1 . , , 1 , , ,

min arg ˆ , ˆ

k l k k l i k k l i i l l MV T l i k k l i i l k k l l l Q T k k l l k

C C C C C C

k

d Q AHf Q f Q f d AH y v K f d AH y v f d AH y K f d AH y D f

D f

λ λ λ − + − − + ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎣ ⎡ ⎟ ⎠ ⎞ ⎜ ⎝ ⎛ − ⎟ ⎠ ⎞ ⎜ ⎝ ⎛ − + − − =

∑ ∑ ∑ ∑

− = − − = −

) (

n n

x f p ∇ =

n n n n

p x x ·

1

λ + =

+

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

SLIDE 31

L.D. Alvarez

Ioannina, September 2004

Resulting for the motion field: And for the HR frame:

( ) ( )

( )

} {

ˆ ˆ ˆ ˆ · ˆ ˆ

2 2 2 1 1 1 1 . , 1 , , 1 , , , 1 m k T T T m k T l i m k k l i i l l MV m k k l l l Q T T T k l m m k m k

C C C C f AH Q Q A H f Q Q f d AH y v K f d AH y K A H d f f

f

λ λ α + + + ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎣ ⎡ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ⎟ ⎠ ⎞ ⎜ ⎝ ⎛ − + − − =

∑ ∑

− = − − +

( ) ( ) ( ) ( )

( )

} {

, 3 3 3 1 , , 1 , , 1 , , , , , , 1 ,

ˆ ˆ ˆ ˆ ˆ · ˆ ˆ

m k l T l i k m k l i i l l MV k m k l l l Q T T m k l k m k l k l m m k l m k l

C C C C d Q Q f d AH y v K f d AH y K A H d f d d d

d

λ α + + ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎣ ⎡ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ⎟ ⎠ ⎞ ⎜ ⎝ ⎛ − + − ∂ ∂ − =

∑

− = − − +

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

SLIDE 32

L.D. Alvarez

Ioannina, September 2004

An estimate for the motion field is first found by

assuming that the high-resolution image is known. Then, estimates for the displacements are found while the high-resolution image is assumed

known. The high-resolution image is then re-

estimated with the recently found displacements, which is then re-estimated with the most recent high-resolution estimate. The process iterates until the estimates converge.

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

SLIDE 33

L.D. Alvarez

Ioannina, September 2004

HR HR simultaneous simultaneous estimation estimation

Idea: To take adventage of the relationship

between the HR images.

In each step we improve the whole sequence. For

the following step we use the improved frames.

We assume that motion vectors have been

previously estimated.

The conversion of a HR frame to its LR and CLR is:

( ) ( )

i i i k i i i k k DCT DCT k

C C T Q T y v y v AHf y

∑ ∑

∀ ∀ −

+ ⎥ ⎦ ⎤ ⎢ ⎣ ⎡ ⎟ ⎠ ⎞ ⎜ ⎝ ⎛ − =

, , 1

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

SLIDE 34

L.D. Alvarez

Ioannina, September 2004

Notation for the whole HR sequence and the whole

CLR sequence:

Conditional distribution for the CLR images given

the HR sequence:

Conditional distribution for each CLR image:

Enforcing similarity between the CLR image and its

corresponding HR image.

( ) ( )

T L T T T L T T

y y y Y f f f F , , , , , ,

2 1 2 1

K K = =

( ) ( )

∏

= =

L i i i L L

p p p f y f f y y F Y | , , | , , ) | (

1 1

K K

( )

{ }

2 4

exp |

i i i i

p AHf y f y − − ∝ λ

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

SLIDE 35

L.D. Alvarez

Ioannina, September 2004

Relationship between HR images: The distribution is:

Including the quality in the prediction (controlled by λ1). Imposing smoothness within the HR images (controlled by λ2). Imposing absence of blocking artifacts (controlled by λ3).

Seeing this relationship it is clear that if one HR

image has changed this will enforce changes in the

ther HR images.

( )

} {

1 2 2 3 1 2 1 2 2 2 , 1 1 1 1

exp , ,

∑ ∑ ∑

= = = − −

− − − − ∝

L i i L i i L i i i i i L

C p AHf Q f Q f d f f f λ λ λ K

( )

k l k k l l , ,

C n f d f + =

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

SLIDE 36

L.D. Alvarez

Ioannina, September 2004

The MAP estimate provide us the framework for

recovering HR information from CLR observations.

Using the Bayesian paradigm the MAP HR

sequence satisfies:

Finally, it remains: To find one solution for the previous equation we

use a ciclyc coordinate descend algorithm. ( ) ( ) { }

F Y F F | max arg ˆ

F

p p =

( )

⎭ ⎬ ⎫ ⎩ ⎨ ⎧ − + ⎭ ⎬ ⎫ ⎩ ⎨ ⎧ + + + − =

∑ ∑ ∑ ∑

= = = = − − L i i i L i L i i L i i i i i i

C

1 2 4 2 1 2 2 3 1 2 1 2 2 , 1 1 1

min arg ˆ AHf y AHf Q f Q f d f F

F

λ λ λ λ

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

SLIDE 37

L.D. Alvarez

Ioannina, September 2004

Observability Observability and and predictibilitly predictibilitly in SR in SR

Idea: There will be useful and useless (concept to

be defined later) pixels in the reconstruction

process. How do we identify them?

Objective: To estimate a HR video sequence from a

CLR video sequence using only observable and predictable pixels in the reconstruction.

Change in our notation:

( )

[ ]

{ }

l l l l

T Q T m m g y + − =

−1

( ) ( )

i i i l i i i l l DCT DCT l

C C T Q T y v y v AHf y

∑ ∑

∀ ∀ −

+ ⎥ ⎦ ⎤ ⎢ ⎣ ⎡ ⎟ ⎠ ⎞ ⎜ ⎝ ⎛ − =

, , 1

CLRl HRk

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

SLIDE 38

L.D. Alvarez

Ioannina, September 2004

SLIDE 39

L.D. Alvarez

Ioannina, September 2004

The system model and the regularization terms are

the same as with the previous model.

The probability distribution of ml, P(ml|fk,dl,k) , is:

P(ml|fk,dl,k)=exp[-½(ml–AHC(dl,k)fk)TKMV,l

1(ml–AHC(dl,k)fk)]

The probability distribution, P(yl|fk,dl,k) , is:

P(yl|fk,dl,k)=exp[-½(yl–AHC(dl,k)fk)T KQ,l

1 (yl–AHC(dl,k)fk)]

The distribution of fk reflects that images are smooth within

homogeneous regions and free of blocking artifacts:

P(fk) = exp [ -½ ( λ1||Q1fk||2 + λ2||Q2AHfk||2 )]

We enforce dl,k to be smooth:

P(dl,k) = exp [ -½ ( λ3||Q3dl,k||2 )]

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

SLIDE 40

L.D. Alvarez

Ioannina, September 2004

Why are not all the pixels valid to reconstruct the

HR image?

We have two different problems:

Observability. Predictibility.

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

SLIDE 41

L.D. Alvarez

Ioannina, September 2004

Observability Observability Problem Problem

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

? yl dl,k fk AHC(dl,k)fk

There are certain regions of yl (or ml) that are not observable from fk via AHC(dl,k)fk. Should we use these zones in our HR reconstruction?

yl -AHC(dl,k)fk ?

SLIDE 42

L.D. Alvarez

Ioannina, September 2004

Predictibility Predictibility problem problem

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

yl fk dl,k AHC(dl,k)fk ? ? yl -AHC(dl,k)fk

If our motion estimation (dl,k) is not good enough there will be a problem with the difference between yl (or ml) and its motion compensation prediction fk (AHC(dl,k)fk). Should we use the corresponding pixels in our HR estimation?

SLIDE 43

L.D. Alvarez

Ioannina, September 2004

Our aim is to determine which pixels of the

compressed LR sequence should be utilized to estimate the HR images.

In the simplest scenario, to estimate the HR k-th

frame we make use of the previous l-th LR image. But there will be regions in the l-th image that will not be observable or predictable from the k-th frame.

Not observable or not predictable pixels do not

provide useful information to the super-resolution method, so they will be removed.

From now on, we will denote observability and

predictibility as observability.

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

SLIDE 44

L.D. Alvarez

Ioannina, September 2004
Procedure to calculate the observability maps:
1. Calculate DFDHR.
2. Obtain ErrorsMapHR that contains ones if DFDAR>T

(and zeros elsewhere).

3. Downsample ErrorsMapHR using a mean filter to
btain a new errors map but in LR (ErrorsMapLR).
4. If a certain pixel in ErrorsMapBR is lower than 0.25 it

will be observable (predictable). ( )(

) ( )

( )

( )(

)

, ,

, , , , , ( , )

l l

l k k l k k y

DFD m n m n C m n T m n

bservable

= − < ⇒

y y

f d f f d f % % % %

( )

( )(

)

, ,

, , ( , ) , , ( , )

l l

l k k l k k m

DFD m n m n m n T m n

bservable

= − < ⇒

m m

f d f f C d f % % % %

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

SLIDE 45

L.D. Alvarez

Ioannina, September 2004

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

1.- DFDHR Difference between the fixed image (k) and the l-th estimated image.

SLIDE 46

L.D. Alvarez

Ioannina, September 2004

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

2.- ErrorMapHR=[DFDHR>Ty ] (ErrorMapHR=[DFDHR>Tm ]) A high resolution pixel with a value greater than a given threshold (Ty or Tm) will not be observable.

SLIDE 47

L.D. Alvarez

Ioannina, September 2004

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

3.- ErrorMapLR =↓[mean(ErrorMapHR)] Use a mean filter on the binary error map and downsample it.

SLIDE 48

L.D. Alvarez

Ioannina, September 2004

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

4.- ObservabilityMap =ErrorMapLR<0.25

SLIDE 49

L.D. Alvarez

Ioannina, September 2004

Once we have calculated the observability maps

the images yl

and ml
are available. These images

come from yl and ml , selecting only the pixels that are useful for the reconstruction.

We had:

And finally:

But we have to introduce here the observability

notion.

P(yl|fk,dl,k)=exp[-½(yl–AHC(dl,k)fk)T KQ,l

1 (yl–AHC(dl,k)fk)]

P(ml|fk,dl,k)=exp[-½(ml–AHC(dl,k)fk)TKMV,l

1(ml–AHC(dl,k)fk)]

P(y,m|fk,d) = Πl [P(yl|fk,dl,k) P(ml|fk,dl,k)] for l=1,2,…,L

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

SLIDE 50

L.D. Alvarez

Ioannina, September 2004

Now, we have

yl

l-th CLR frame using only observable pixels.

ml

observable pixels in the prediction, obtained via

motion compensation from yl

.

If yl is now yl

the difference yl–AHC(dl,k)fk will be

yl

–Al
HC(dl,k)fk .

Analogously, from ml–AHC(dl,k)fk we have

ml

–Al
HC(dl,k)fk.

Al

matrices are different due to the observability

maps.

The covariance matrices are also differents:

KQ,l

1 (KQ,l
)-1 .

KMV,l

1 (KMV,l
)-1 .

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

SLIDE 51

L.D. Alvarez

Ioannina, September 2004

Including the observability maps, the probability

distributions become: P(yl

|fk,dl,k) =

exp[-½(yl

– Al
HC(dl,k)fk)T (KQ,l
)-1 (yl
– Al
HC(dl,k)fk)]

P(ml

|fk,dl,k) =

exp[-½(ml

– Al
HC(dl,k)fk)T ( KMV,l
)-1 (ml
– Al
HC(dl,k)fk)]

Our aim is to estimate the high resolution image fk

and high resolution motion vectors, d, satisfying:

( ) (

)

[ ]

d f m y d f d f

d f

, | , , max arg ˆ , ˆ

, k

k

k

P P

k

=

INTRODUCTION

Myself
University
Research group

RESEARCH

Introduction

and notation

System model
Problem

formulation

Optimization

procedure

HR simultaneous

estimation

Observability

& predictability

Results

SLIDE 52

L.D. Alvarez

Ioannina, September 2004

EXPERIMENTAL RESULTS EXPERIMENTAL RESULTS

SLIDE 53

L.D. Alvarez

Ioannina, September 2004

Sequence: Mobile. Image size: 704x576 – Cropped to a size of

176x144.

Decimation factor: 2. Compression: MPEG-4 at 1024Kbps. Parameters:

Q1, Q3 : 3x3 discrete Laplacian. Q2 : Difference accross the horizontal and vertical block boundaries. λ1=100 λ2=0.01 λ3=0.002 λ4=1 αfl=0.125

Stop criteria:

7 2 2 1

10 1 ˆ / ˆ ˆ

− +

< − x

k k k

F F F

SLIDE 54

L.D. Alvarez

Ioannina, September 2004

Reconstruction example by the basic procedure:

a) HR original image (cropped region). b) CLR obtained by bilinear interpolation. c) Reconstruction obtained using the proposed method. a) c) (33.2 dB) b) (30.5 dB)

SLIDE 55

L.D. Alvarez

Ioannina, September 2004

Reconstruction example by the simultaneous HR procedure:

a) HR original image (cropped region). b) CLR obtained by bilinear interpolation. c) Reconstruction obtained using the proposed method. a) c) (33.0 dB) b) (30.5 dB)

SLIDE 56

L.D. Alvarez

Ioannina, September 2004

Consecutive CLR frames (numbers: 8, 9, 10) of the sequence.

HR frame number 9. Observable pixel maps for the CLR frames and LR motion compensated numbers 7, 8, 10 and 11 using threshold Ty=16.

SLIDE 57

L.D. Alvarez

Ioannina, September 2004

Bilinear interpolation

f frame 9.

HR estimation without

bservability maps

(fig. A). HR estimation using

bservability maps

(fig. B). Two details of image in fig. A. Two details of image in fig. B

SLIDE 58

L.D. Alvarez

Ioannina, September 2004

Reconstruction example using

bservability maps:

a) HR original image (cropped region). b) CLR obtained by bilinear interpolation. c) Reconstruction obtained using the proposed method. a) c) (33.7 dB) b) (30.5 dB)

SLIDE 59

L.D. Alvarez

Ioannina, September 2004

PSNR evolution using only

bservability map for yl with

different threshold values. PSNR evolution using

bservability maps for yl and

ml with different threshold values.

SLIDE 60

L.D. Alvarez

Ioannina, September 2004

Thank Thank you you very very much much for for coming coming and and for for your your interest interest. .

SLIDE 61

L.D. Alvarez

Ioannina, September 2004

m d k l m d m k l m k l

p d d

, , , , 1 ,

· ˆ ˆ α − =

+

( ) ( ) [ ]

[ ] [ ]

[ ]

2 , 3 3 2 2 2 2 1 1 1 ) ( ) ( 1 , 1 ) ( ) ( ) ( 1 , ) ( ,

· · · · · · · · · · · · · · · · · · · · · · · · · · ,

. , . , , ,

k l k k l i k d i v l MV t l i k d i v k d l l Q t k d l m d

d Q d f H A Q d f Q d f C H A y C K f C H A y C d f C H A y K f C H A y d d d f p

k l i l k l i l k l k l

λ λ λ ∂ ∂ + ∂ ∂ + ∂ ∂ + + ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎣ ⎡ ⎟ ⎠ ⎞ ⎜ ⎝ ⎛ − ⎟ ⎠ ⎞ ⎜ ⎝ ⎛ − ∂ ∂ + + − − ∂ ∂ = = ∂ ∂ =

∑ ∑

− = − − = −

) (

n n

x f p ∇ =

n n n n

p x x ·

1

λ + =

+

( ) ( )

} {

, 3 3 3 1 ) ˆ ( ) ( 1 , ) ˆ ( 1 , , ) ˆ ( , , , 1 ,

ˆ · · · · · · · · · · · · · · · · ˆ · · ˆ ˆ

, , , ,

m k l t l i k d i v l MV k d l l Q t t m k l k d k l m d m k l m k l

d Q Q f C H A y C K f C H A y K A H d f C d d

m k l i l m k l m k l

λ α + ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎣ ⎡ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ⎟ ⎠ ⎞ ⎜ ⎝ ⎛ − + − ∂ ∂ − =

∑

− = − − +

SLIDE 62

L.D. Alvarez

Ioannina, September 2004

m f m f m k m k

p f f

, , 1

· ˆ ˆ α − =

+

( ) ( ) [ ]

[ ] [ ]

[ ]

2 , 3 3 2 2 2 2 1 1 1 ) ( ) ( 1 , 1 ) ( ) ( ) ( 1 , ) ( ,

· · · · · · · · · · · · · · · · · · · · · · · · · · ,

. , . , , ,

k l k k l i k d i v l MV t l i k d i v k d l l Q t k d l m f

d Q f f H A Q f f Q f f C H A y C K f C H A y C f f C H A y K f C H A y f f d f p

k l i l k l i l k l k l

λ λ λ ∂ ∂ + ∂ ∂ + ∂ ∂ + + ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎣ ⎡ ⎟ ⎠ ⎞ ⎜ ⎝ ⎛ − ⎟ ⎠ ⎞ ⎜ ⎝ ⎛ − ∂ ∂ + + − − ∂ ∂ = = ∂ ∂ =

∑ ∑

− = − − = −

( ) ( )

} {

ˆ · · · · · · · ˆ · · · ˆ · · · · · ˆ · · · · · · · · · ˆ ˆ

2 2 2 1 1 1 1 ) ( ) ( 1 , ) ( 1 , ) ( , 1

. , , ,

m k t t t m k t l i m k d i v l MV m k d l l Q t t TF k TB k l t d m f m k m k

f H A Q Q A H f Q Q f C H A y C K f C H A y K A H C f f

k l i l k l k l

λ λ α + + + ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎣ ⎡ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ⎟ ⎠ ⎞ ⎜ ⎝ ⎛ − + − − =

∑ ∑

− = − − + − = +

) (

n n

x f p ∇ =

n n n n

p x x ·

1

λ + =

+

SLIDE 63

L.D. Alvarez

Ioannina, September 2004

Quantizers (axb) Quantizer Variance 8x8 block Quantizer Variance block (64x64) diag T · QVb · T-1 Covariance Matrix block (64x64) Covariance Matrix (a/8)·64x(b/8)·64

How do we construct the covariance matrix.

SLIDE 64

L.D. Alvarez

Ioannina, September 2004

The performance of the proposed algorithm was

evaluated by measuring the peak signal-to-noise ratio.

PSNR is calculated as follows:

r, following our notation, as:

2 2 10