Optimising Data for PDE-Based Inpainting and Compression Laurent - - PowerPoint PPT Presentation

Dagstuhl Seminar Inpainting-Based Image Compression

Optimising Data for PDE-Based Inpainting and Compression

Laurent Hoeltgen

hoeltgen@b-tu.de

Chair for Mathematics of Engineering & Numerical Optimisation Brandenburg University of Technology, Cottbus - Senftenberg

November 17th, 2016

This work is licensed under a Creative Commons “Attribution-ShareAlike 4.0 International” license.

Motivation (1)

PDE-Based Image Compression

alternative to transform based approaches has three important pillars each requires advanced optimisation

Data Selection Data Storage Data Reconstruction Image Compression

Motivation (2)

Challenges in Mathematical Modelling

We have to take a few tough decisions:

How to reconstruct the image?

PDE has significant influence on our design principles

What are our optimisation criteria?

Which data should we optimise? Should we maximise the quality or minimise the file size?

Outlook

• Data Reconstruction
• Data Selection
• Data Selection in the Domain
• Data Selection in the Co-Domain
• Summary and Conclusion

Data Reconstruction (1)

PDE-Based Image Inpainting

We could use any diffusion type PDE. However, we want it to be:

simple to analyse

linear, parameter free, ...

applicable to any domain and co-domain

no restriction on image type, size, quantisation, ...

fast to carry out

Laplace interpolation fulfils all these requirements.

Data Reconstruction (2)

Laplace Interpolation (Noma, Misuglia, 1959)

consider Laplace equation with mixed boundary conditions:

Ω ΩK ΩK ∂Ω 8 > < > : −∆u = 0,

• n Ω

u = g,

• n ΩK

∂nu = 0,

• n ∂Ω

ΩK: represents known data Ω \ ΩK: region to be inpainted (i.e. missing data) image reconstructions given as solutions u

Data Selection (1)

Highest Accuracy or Smallest Memory Footprint?

Optimisation strategies can be split into 2 groups:

• 1. maximise accuracy at the expense of file size

data locations have no apparent structure data values are real valued large amounts of data are preferred

• 2. minimise memory footprint at the expense of accuracy

data positions stored in memory efficient structures data values are quantised sparse sets of data are preferred

Data Selection (2)

Finding Optimal Interpolation Data

We can’t optimise accuracy and file size at the same time.

Using all image data yields perfect accuracy. Using no image data at all yields perfect file size.

Our Strategy:

We target a fixed amount of image data and optimise accuracy. In case of identical accuracy, we chose the one with smaller size.

Data Selection (3)

brute force search of optimal pixel locations is impossible 105600 ways to choose 5% of data from 0.07 megapixel image most smart-phones yield 12 megapixel photographs for comparison:

1023 × =

Data Selection in the Domain (1)

Laplace Interpolation (Reminder)

Ω ΩK ΩK ∂Ω 8 > < > : −∆u = 0,

• n Ω \ ΩK

u = g,

• n ΩK

∂nu = 0,

• n ∂Ω

can be rewritten as (Mainberger et al. 2011): ( c (u − g) + (1 − c) (−∆)u = 0,

• n Ω

∂nu = 0,

• n ∂Ω

with c ≡ 1 on ΩK and c ≡ 0 on Ω \ ΩK

Data Selection in the Domain (2)

Regularised Laplace Interpolation

( c (u − g) + (1 − c) (−∆)u = 0,

• n Ω

∂nu = 0,

• n ∂Ω

PDE makes also sense if c : Ω → R

can be seen as regularisation

Regions with c > 1 violate Max-Min principle

PDE resembles backward diffusion process may also be interpreted as a Helmholtz equation contrast enhancement

unconditional existence of a solution difficult to show

Data Selection in the Domain (3)

An Optimal Control Model for Sparse Masks

(H., Setzer, Weickert 2013)

• ptimal non-binary and sparse masks c given by

arg min

u, c

Z

Ω

1 2 (u − g)2 + λ|c| + ε 2|c|2 dx ff subject to: ( c (u − g) + (1 − c)(−∆)u = 0,

• n Ω

∂nu = 0,

• n ∂Ω

model has strong similarities to Belhachmi et al. 2009

• 1

2 (u − g)2 favours accurate reconstructions λ|c| favours sparse sets of data

• ε

2|c|2 required for technical reasons PDE enforces feasible solutions

Data Selection in the Domain (4)

Interpretation

arg min

u, c

Z

Ω

1 2 (u − g)2 + λ|c| + ε 2|c|2 dx ff subject to: ( c (u − g) + (1 − c)(−∆)u = 0,

• n Ω

∂nu = 0,

• n ∂Ω

energy reflects trade-off between accuracy and sparsity

• bjectives cannot be fulfilled simultaneously

λ steers sparsity of the interpolation data

small, positive λ: many data points, but good reconstruction large, positive λ: few data points, but bad reconstruction

non-convex, non-smooth, and large scale optimisation

Data Selection in the Domain (5)

A Solution Strategy

linearise constraint to handle non-convexity:

T(u, c) := c (u − g) + (1 − c)(−∆)u T(u, c) ≈ T(uk, ck) + DuT(uk, ck)(u − uk) + DcT(uk, ck)(c − ck)

add proximal term and solve iteratively

arg min

u, c

Z

Ω

1 2 (u − g)2 + λ|c| + ε 2|c|2 + µ 2 “ u − uk”2 + µ 2 “ c − ck”2 dx ff T(uk, ck) + DuT(uk, ck)(u − uk) + DcT(uk, ck)(c − ck) = 0 until fixed point is reached

Data Selection in the Domain (6)

Theoretical Findings

algorithm yields several interesting results:

• 1. energy decreasing as long as:

1 2 “ uk+1 − g2

2 − uk − g2 2

” 6 λ “ ck+11 − ck1 ” + ε 2 “ ck+12

2 − ck2 2

” gain in sparsity must outweigh loss in accuracy

• 2. fixed-points fulfil necessary optimality conditions:

u − g − DuT(u, c)⊤p = 0 λ∂ (·1) (c) + εc + DcT(u, c) p ∋ 0 T(u, c) = 0

Data Selection in the Domain (7)

Example (Input)

Data Selection in the Domain (8)

Evolution of the Iterates

100 200 300 400 500 600 700 iteration energy mean squared error

Data Selection in the Domain (8)

Evolution of the Iterates

100 200 300 400 500 600 700 iteration energy mean squared error

Data Selection in the Domain (8)

Evolution of the Iterates

100 200 300 400 500 600 700 iteration energy mean squared error

Data Selection in the Domain (8)

Evolution of the Iterates

100 200 300 400 500 600 700 iteration energy mean squared error

Data Selection in the Domain (9)

Example (5% of Mask Points and Reconstruction)

Data Selection in the Co-Domain (1)

Tonal Optimisation

• ptimal pixel values necessary to maximise quality

quantisation to n ≪ 256 colours essential for compression

best number of colours hard to determine

• ften 30% less memory needed
• ptimisation criteria should take file size into account

hard to predict

Data Selection in the Co-Domain (2)

The Inpainting Operator

given mask c, solution u = u(c) of

c (u − g) + (1 − c) (−∆)u = 0,

• n Ω

∂nu = 0,

• n ∂Ω

can be expressed in terms of a linear inpainting operator Mc: u = Mc(cg)

• ptimal data values can be found by solving:

arg min

x

Z

Ω

|Mc(cx) − f |2 dx ff

Data Selection in the Co-Domain (3)

Continuous Tonal Optimisation

least squares model suggested by Mainberger et al. 2011 yields arbitrary values in R equivalence to mask optimisation

• ptimal non-binary masks c equivalent to
• ptimal colours in R with binary masks

(H., Weickert 2015)

maximises reconstruction quality storage of colour values in R is very expensive

Data Selection in the Co-Domain (4)

Remarks

The presented model yields high quality data. The optimisation is time consuming. Storing this data as-is is too expensive for compression tasks. We need algorithms to simplify the data:

memory efficient representation of arbitrary binary masks good colour quantisation strategies

Data Selection in the Co-Domain (5)

Quantisation of Optimal Data Values

Optimal colour values gather around few dominant colours. replacing all colours by their closest dominant colour yields:

reconstruction error becomes slightly larger data becomes much easier to compress

achievable by using clustering strategies

Important questions:

Which clustering method is the best? How to chose our feature values?

Data Selection in the Co-Domain (6)

Distribution of Optimal Colour Values

• ptimal colour values of the trui test image on mask positions

64 128 192 20 40 60

Data Selection in the Co-Domain (7)

Evaluated Strategies

we tested with various feature choices:

k-means++ hierarchical clustering Gaussian mixture models

Findings: k-means++ on colour values at mask positions works quite well

Data Selection in the Co-Domain (8)

Discrete/Continuous Tonal Optimisation

(H., Breuß 2016)

apply k-means++ on colour values from mask locations

10 15 20 25 30 35 40 45 50 55 60 65 70 75 45 50 55 number of clusters mean squared error

hierarchical k-means probabilistic no optimisation

k-means++ 30 colours outperforms original data (170 colours)

• ptimal quantisation found with silhouette statistics

Data Selection in the Co-Domain (9)

Silhouette plots (clustering with 30 clusters)

Silhouette plots allow to evaluate clustering results. Each bar represents a feature.

Values close to 1 indicate a correct labelling. Values close to -1 are probably labelled false.

we obtained:

−1 −0.5 0.5 1 Hierarchical Mean: 0.59, Median: 0.8 −1 −0.5 0.5 1 k-means++ Mean: 0.63, Median: 0.73

Data Selection in the Co-Domain (10)

Remarks

proposed clustering optimises reconstruction error distribution might be unsuitable for data compression final compressed file size is very hard to predict efficient clustering based on file size is difficult (but necessary)

Summary and Conclusion

Take Home Message

PDE-based compression requires optimisation at all levels.

• ptimal models depend on choice of the PDE

trade-off between quality and file size necessary file compression add further constraints some tasks remain difficult to solve

Thank You

Thank you very much for your attention!

More information at:

http://www-user.tu-cottbus.de/~hoeltgen/