High Quality Image Compression Using PDE-Based Inpainting with - - PowerPoint PPT Presentation

2016 SIAM Conference on Imaging Science Albuquerque, May 23 – May 26

High Quality Image Compression

Using PDE-Based Inpainting with Optimal Data Laurent Hoeltgen

hoeltgen@b-tu.de

Chair for Applied Mathematics Brandenburg Technical University Cottbus - Senftenberg

May 23, 2016

This work is licensed under a Creative Commons “Attribution-ShareAlike 4.0 International” license.

Motivation

PDE-Based Image Compression

alternative to transform based approaches has 3 important pillars each requires advanced optimisation

Motivation

PDE-Based Image Compression

alternative to transform based approaches has 3 important pillars each requires advanced optimisation

Data Selection

Motivation

PDE-Based Image Compression

alternative to transform based approaches has 3 important pillars each requires advanced optimisation

Data Selection Data Storage

Motivation

PDE-Based Image Compression

alternative to transform based approaches has 3 important pillars each requires advanced optimisation

Data Selection Data Storage Data Reconstruction

Motivation

PDE-Based Image Compression

alternative to transform based approaches has 3 important pillars each requires advanced optimisation

Data Selection Data Storage Data Reconstruction Image Compression

Outlook

• Data Reconstruction

PDE-Based Image Inpainting

• Data Selection

Optimal Data Locations Optimal Data Values

• Data Storage
• Summary and Conclusion

Data Reconstruction (1)

Data Reconstruction

We could use any diffusion type PDE. However, we want it to be:

simple to analyse

linear, parameter free, ...

applicable for any domain and codomain

no restriction on image type, size, quantisation, ...

fast to carry out

Laplace interpolation fulfils all these requirements.

Data Reconstruction (2)

Laplace Interpolation for Image Inpainting

consider Laplace equation with mixed boundary conditions:

Ω ΩK ΩK ∂Ω 8 > < > : −∆u = 0,

• n Ω

u = g,

• n ΩK

∂nu = 0,

• n ∂Ω

ΩK: represents known data Ω \ ΩK: region to be inpainted (i.e. missing data) image reconstructions given as solutions u

Data Selection (1)

Finding Optimal Interpolation Data

brute force search of optimal pixel locations is impossible 105600 ways to choose 5% of data from 0.07 megapixel image for comparison:

1023 × =

Data Selection (2)

Alternatives

search space reduction with binary trees

(Galić et al. 2005, Schmaltz et al. 2009)

analytical derivation

(Belhachmi et al. 2008)

probabilistic search methods

(Mainberger et al. 2012)

stochastic optimisation on binary trees

(Peter et al. 2015)

regularisation and optimal control models

(Hoeltgen et al., 2013)

Data Selection (3)

Search Space Reduction with Binary Trees

iteratively refine block-wise decomposition a block is split if error is too large interpret refinements as nodes of a binary tree

Evaluation: average reconstruction quality run times of few seconds at most storing data locations is extremely cheap

Data Selection (4)

Binary Tree Representation

1 2 4 5 6 7 3

Data Selection (4)

Binary Tree Representation

1 1 2 4 5 6 7 3

Data Selection (4)

Binary Tree Representation

1 2 3 1 2 4 5 6 7 3

Data Selection (4)

Binary Tree Representation

1 2 3 4 5 2 1 2 4 5 6 7 3

Data Selection (4)

Binary Tree Representation

1 2 3 4 5 2 6 7 2 5 1 2 4 5 6 7 3

Data Selection (5)

Example

Data Selection (6)

Analytical Derivation

derive optimality conditions in Sobolev spaces

density of optimal data proportional to Laplacian magnitude

apply dithering to get discrete distribution

works best with electrostatic halftoning

(Schmaltz et al. 2010)

Floyd Steinberg is a faster alternative

Evaluation: average reconstruction quality capable of real time processing (30 fps) reconstruction data difficult to store

Data Selection (7)

Example

Data Selection (7)

Example

Data Selection (8)

Probabilistic Search Methods

iteratively test importance of random subsets of pixels

discard unimportant pixels shuffle remaining candidates

similarities to genetic algorithms and simulated annealing works for any PDE, but only in discrete settings

Evaluation: very good reconstruction quality extremely slow: run times of days to weeks reconstruction data difficult to store

Data Selection (9)

Example

Data Selection (10)

Laplace Interpolation (Reminder)

Ω ΩK ΩK ∂Ω 8 > < > : −∆u = 0,

• n Ω \ ΩK

u = g,

• n ΩK

∂nu = 0,

• n ∂Ω

can be rewritten as ( c (u − g) + (1 − c) (−∆)u = 0,

• n Ω

∂nu = 0,

• n ∂Ω

with c ≡ 1 on ΩK and c ≡ 0 on Ω \ ΩK

Data Selection (11)

Regularisation and Optimal Control Models

PDE makes also sense if c : Ω → R (≈ regularisation)

• ptimal non-binary and sparse masks c given by

arg min

u, c

Z

Ω

1 2 (u − g)2 + λ|c| + ε 2|c|2 dx ff subject to: ( c (u − g) + (1 − c)(−∆)u = 0,

• n Ω

∂nu = 0,

• n ∂Ω

Evaluation: highest quality of all presented methods slow: run times of hours to days reconstruction data difficult to store

Data Selection (12)

Example

Data Selection (13)

Tonal Optimisation

• ptimal pixel values necessary to maximise quality

quantisation to n ≪ 256 colours essential for compression

best number of colours hard to determine

• ften 30% less memory needed
• ptimisation criteria should take file size into account

hard to predict

Data Selection (14)

Continuous Tonal Optimisation

• ptimise values in R

Mainberger et al. (2011): least squares model Hoeltgen et al. (2015): equivalence to mask optimisation

• ptimal non-binary masks c equivalent to
• ptimal colours in R with binary masks!

maximises reconstruction quality storage of colour values in R is very expensive

Data Selection (15)

Discrete Tonal Optimisation

Schmaltz et al. (2009) iteratively optimise colour mapping

uniform distribution of colours necessary very slow number of colours fixed beforehand excellent quality

Extension to take file size into account by Peter (2014) k-means based alternative by Hoeltgen et al. (2016)

arbitrary distribution of colours possible much faster allows optimisation of quantisation levels doesn’t consider file size

Data Selection (16)

Discrete/Continuous Tonal Optimisation

apply K-means on colour values from mask locations

10 15 20 25 30 35 46 48 50 52 number of clusters mean squared error K-means

• riginal data

K-means 30 colours outperforms original data (170 colours)

• ptimal quantisation found with silhouette statistics

Data Storage (1)

Flexible Codec Design

depending on the application:

• 1. preserve quality at the expense of larger file size

arbitrary mask positions tonal optimisation of data and binarisation of mask

• 2. sacrifice accuracy to maximise compression rate

use binary tree representation for mask tonal optimisation

Data Storage (2)

Image Encoding and Decoding

Encoding:

• 1. computation of a binary mask (any strategy)
• 2. computation of optimal grey values (with quantisation)
• 3. encoding of all the data in a container file
• 4. application of a high performing entropy coder (PAQ)

Decoding:

reverse order with additional final inpainting can be done in real time (30 fps) (Baum 2013)

independent from chosen optimisation strategies!

Data Storage (3)

Comparison to JPEG and JPEG 2000

10 : 1 15 : 1 20 : 1 10 20 30 compression rate mean squared error exact mask tree mask JPEG JPEG 2000

Data Storage (3)

Comparison to JPEG and JPEG 2000

10 : 1 15 : 1 20 : 1 10 20 30 compression rate mean squared error exact mask tree mask JPEG JPEG 2000

Data Storage (3)

Comparison to JPEG and JPEG 2000

10 : 1 15 : 1 20 : 1 10 20 30 compression rate mean squared error exact mask tree mask JPEG JPEG 2000

Summary and Conclusion

Take Home Message

PDE-based compression is a viable alternative:

applicable to any data (images, videos, hyper-spectral, ...) user friendly (single parameter for quality setting) well suited if encoding time does not matter mathematically well founded

Thank You

Thank you very much for your attention!

More information at:

http://www-user.tu-cottbus.de/~hoeltgen/

References

Laurent Hoeltgen and Michael Breuß. “Efficient Co-Domain Quantisation for PDE-based Image Compression”. Proceedings

• f the Conference Algoritmy. 2016.

Pascal Peter et al. “Evaluating the True Potential of Diffusion-based Inpainting in a Compression Context”. Signal Processing: Image Communication (2016). Laurent Hoeltgen and Joachim Weickert. “Why Does Non-binary Mask Optimisation Work for Diffusion-based Image Compression?” Energy Minimization Methods in Computer Vision and Pattern Recognition. 2015. Kevin Baum. “GPGPU Supported Diffusion-based Naive Video Compression”. Master’s Thesis. Saarland University, 2013.

References

Laurent Hoeltgen, Simon Setzer, and Joachim Weickert. “An Optimal Control Approach to Find Sparse Data for Laplace Interpolation”. Energy Minimization Methods in Computer Vision and Pattern Recognition. 2013. Markus Mainberger et al. “Optimising Spatial and Tonal Data for Homogeneous Diffusion Inpainting”. Scale Space and Variational Methods in Computer Vision. 2012.

• Z. Belhachmi et al. “How to Choose Interpolation Data in

Images”. SIAM Journal on Applied Mathematics (2009). Christian Schmaltz, Joachim Weickert, and Andres Bruhn. “Beating the Quality of JPEG 2000 with Anisotropic Diffusion”. Pattern Recognition. 2009.

References

Irena Galic et al. “Towards PDE-based Image Compression”. Variational, Geometric, and Level Set Methods in Computer Vision. 2005.

Related Work

Previous findings on PDE-Based inpainting and compression:

Contour-based reconstructions (Carlsson 1988) Wavelet decompositions with TV (Chan et al. 2001) Adaptive triangulations with Splines (Demaret et al. 2004) Toppoints in scale space (Kanter et al. 2005) Laplacian approximations for elevation maps (Xie et al. 2006) Spline based representations (Orzan et al. 2008) Bilevel optimisation (Chen et al. 2014)

Properties of Discretised PDE

Standard finite difference schemes in 2D for c (u − g) + (1 − c) (−∆)u = 0,

• n Ω

∂nu = 0,

• n ∂Ω

yield a linear system of equations. The system matrix has the following properties:

• 1. All eigenvalues are real if c is bounded by 1.
• 2. Matrix is invertible if c maps to

ˆ 0, 8

7

˜ .

• 3. Solutions obey max-min principle if c maps to [0, 1].

Generalisation of Mainberger et al. (2011) to the non-binary case.

PDE-Based Inpainting with Sparse Data (5%)

reconstruction quality depends on data selection

PDE-Based Inpainting with Sparse Data (5%)

reconstruction quality depends on data selection

PDE-Based Inpainting with Sparse Data (5%)

reconstruction quality depends on data selection

Binary Tree Coding

1 1 1 1 1 1 1 1 1 Binary encoding = 0 1 1 1 | {z }

Level 3

0 0 1 0 1 1 | {z }

Level 4

Range Coding: {3, 4}

Interpretation of the Optimal Control Model

arg min

u, c

Z

Ω

1 2 (u − g)2 + λ|c| + ε 2|c|2 dx ff

energy reflects trade-off between accuracy and sparsity

• bjectives cannot be fulfilled simultaneously

data term enforces accuracy sparsity term prefers sparse masks

λ steers sparsity of interpolation data

small, positive λ: many data points, good reconstruction large, positive λ: few data points, bad reconstruction

Benefits of a Non-Binary Mask c

Finding binary masks is a non-convex, combinatorial task. Non-binary masks allow better fits to the data. 1 4 1 2 3 4

1

4 3

• riginal

reconstruction mask c Reconstruction by piecewise linear inerpolation in 1D.

Benefits of a Non-Binary Mask c

Finding binary masks is a non-convex, combinatorial task. Non-binary masks allow better fits to the data. 1 4 1 2 3 4

1

4 3

• riginal

reconstruction mask c L2 Error: 0.45

Benefits of a Non-Binary Mask c

Finding binary masks is a non-convex, combinatorial task. Non-binary masks allow better fits to the data. 1 4 1 2 3 4

1

4 3

• riginal

reconstruction mask c L2 Error: 0.43

Benefits of a Non-Binary Mask c

Finding binary masks is a non-convex, combinatorial task. Non-binary masks allow better fits to the data. 1 4 1 2 3 4

1

4 3

• riginal

reconstruction mask c L2 Error: 0.41

Benefits of a Non-Binary Mask c

Finding binary masks is a non-convex, combinatorial task. Non-binary masks allow better fits to the data. 1 4 1 2 3 4

1

4 3

• riginal

reconstruction mask c L2 Error: 0.39

Benefits of a Non-Binary Mask c

Finding binary masks is a non-convex, combinatorial task. Non-binary masks allow better fits to the data. 1 4 1 2 3 4

1

4 3

• riginal

reconstruction mask c L2 Error: 0.37

Benefits of a Non-Binary Mask c

Finding binary masks is a non-convex, combinatorial task. Non-binary masks allow better fits to the data. 1 4 1 2 3 4

1

4 3

• riginal

reconstruction mask c L2 Error: 0.34

A Solution Strategy

Linearise constraint to handle non-convexity:

T(u, c) := c (u − g) + (1 − c)(−∆)u T(u, c) ≈ T(uk, ck) + DuT(uk, ck)(u − uk) + DcT(uk, ck)(c − ck)

Add proximal term and solve iteratively

arg min

u, c

Z

Ω

1 2 (u − g)2 + λ|c| + ε 2|c|2 + µ 2 “ u − uk”2 + µ 2 “ c − ck”2 dx ff T(uk, ck)+DuT(uk, ck)(u−uk)+DcT(uk, ck)(c−ck) = 0 until fixed point is reached.

Algorithmic Details

approach is similar to

LCL algorithm (Murthagh et al. 1982), EM/MM method (Orthega et al. 1970).

Linearised problem is convex and easier to solve. Derivation of the conjugate dual problem is possible.

unconstrained convex optimisation problem solvable via gradient descent may yield more accurate solutions and faster convergence

Theoretical Findings

algorithm yields several interesting results:

• 1. energy decreasing as long as:

1 2 “ uk+1 − g2

2 − uk − g2 2

” 6 λ “ ck+11 − ck1 ” + ε 2 “ ck+12

2 − ck2 2

” gain in sparsity must outweigh loss in accuracy

• 2. fixed-points fulfil necessary optimality conditions:

u − g − DuT(u, c)⊤p = 0 λ∂ (·1) (c) + εc + DcT(u, c) p ∋ 0 T(u, c) = 0

Evolution of the Iterates

100 200 300 400 500 600 700 Iteration energy mean squared error

Evolution of the Iterates

100 200 300 400 500 600 700 Iteration energy mean squared error

Evolution of the Iterates

100 200 300 400 500 600 700 Iteration energy mean squared error

Evolution of the Iterates

100 200 300 400 500 600 700 Iteration energy mean squared error

The Inpainting Operator

standard finite difference schemes in 2D for c (u − g) + (1 − c) (−∆)u = 0,

• n Ω

∂nu = 0,

• n ∂Ω

yield a linear system diag(c) + (1 − diag(c)) (−∆) | {z }

=:A(c)

u = diag(c)g the matrix M(c) := A(c)−1diag(c) is called inpainting operator

Continuous Tonal Optimisation

let mask c be fixed and M(c) the inpainting operator. Mainberger et al. (2012) suggest to optimise the grey values g: Continuous Tonal Optimisation f = arg min

x∈Rn

1 2M (c) x − g2

2

ff

linear least squares problem efficient solvers exist

LSQR (Paige & Saunders, 1982) based solver for CPUs primal dual (Chambolle & Pock, 2011) solver for GPUs

Mask VS. Grey Value Optimisation

Optimising grey values and mask values can give the same results.

1 2

1

4 3

Mask Value Optimisation

1 2

1

4 3

Grey Value Optimisation

An Equivalence Result

complex relationship between inpainting data g and mask c:

c (u − g) + (1 − c)(−∆)u = 0,

• n Ω

∂nu = 0,

• n ∂Ω

Important: mask locations coincide with data locations.

Equivalent Data Optimisation If mask positions are fixed by set K: mask value optimisation is equivalent to grey value optimisation min

ci, i∈K

n M (c) g − g2

2

• = min

xi, i∈K

n M (¯ c) x − g2

2

• with ¯

c being binary mask for K.