Multi-Modal Image Processing with Applications to Art Investigation - - PowerPoint PPT Presentation
Multi-Modal Image Processing with Applications to Art Investigation and Beyond Miguel Rodrigues Dept. Electronic and Electrical Engineering University College London Collaborators Ingrid Daubechies Bruno Cornellis Duke U. VUB Pingfan Song
Joao Mota Heriot Watt U. Nikos Deligiannis VUB
Ingrid Daubechies Duke U. Bruno Cornellis VUB Pingfan Song UCL
T1 and T2 MRI and PET
The questions that arise in medical imaging include:
across the various imaging modalities?
image modalities?
LIDAR Data Hyper-Spectral Data
SAR Data
The questions that arise in remote sensing also include:
across the various imaging modalities?
image modalities?
circles to rub out an earlier piece of writing by means of washing or scraping the manuscript, in order to prepare it for a new text.
interested in older writings, so multi-modal data processing technology is needed to attempt to recover erased old texts.
Palimpsest contains a Cyrillic overwriting and partly Greek, partly Cyrillic underwritings, which have been washed off
The Ghent Alterpiece - Visuals The Ghent Alterpiece – X-Rays
Some tasks that arise in art investigation, restoration and preservation include:
layers for technical study purposes.
degradation / restoration. The imaging modalities used in art investigation include macrophotography, X-radiography, hyperspectral imaging, infrared imaging, X-ray fluorescence (XRF) mapping
X-ray radiation transmission radiograph (XRR) Infrared reflectograph (IRR)
Dik et al. Visualization of a Lost Painting by Vincent van Gogh Using Synchrotron Radiation Based X-ray Fluorescence Elemental Mapping. Anal. Chem. 2008, 80, 6436–6442
dictionary sparse vector
Ψ 𝑦 𝑨
data vector
𝑦 = Ψ𝑨 + 𝑥
The data vector 𝑦 ∈ ℝ) can be represented in terms of a sparse vector 𝑨 ∈ ℝ* as follows: +
noise vector
𝑥
where Ψ ∈ ℝ)×* is a dictionary such as a wavelet basis or a learnt one.
dictionary sparse vector
Ψ 𝑦 𝑨
data vector
𝑦 = Ψ𝑨 + 𝑥
The data vector 𝑦 ∈ ℝ) can be represented in terms of a sparse vector 𝑨 ∈ ℝ* as follows: +
noise vector
𝑥
where Ψ ∈ ℝ)×* is a dictionary such as a wavelet basis or a learnt one.
dictionary sparse vector
Ψ 𝑦 𝑨
data vector
𝑦 = Ψ𝑨 + 𝑥
The data vector 𝑦 ∈ ℝ) can be represented in terms of a sparse vector 𝑨 ∈ ℝ* as follows: +
noise vector
𝑥
where Ψ ∈ ℝ)×* is a dictionary such as a wavelet basis or a learnt one.
dictionary sparse vector
Ψ 𝑦 𝑨
data vector
𝑦 = Ψ𝑨 + 𝑥
The data vector 𝑦 ∈ ℝ) can be represented in terms of a sparse vector 𝑨 ∈ ℝ* as follows: +
noise vector
𝑥
Sparse representations have had implications in various problems such as:
where Ψ ∈ ℝ)×* is a dictionary such as a wavelet basis or a learnt one.
Sparse Vector (z) Recovered Sparse Vector Measured Vector (y)
The measurement vector is generated from the signal vector as follows: The signal sparse representation vector can be recovered from the measurement vector as follows: y= Φ𝑦 = ΦΨ𝑨 𝑨̂ = arg min
4
𝑨 5subject to 𝑧 = ΦΨ𝑨 where is a “wide” measurement matrix. Φ Optimization- and greedy-based algorithms can be used to reconstruct the signal vector from the measurement vector.
Noisy Image De-Noised Image
Blurred Image De-Blurred Image
Original Image New Image
One postulates that the true image admits a sparse representation in some dictionary.
One then obtains the sparse represent. associated with the image as well as the dictionary given the noisy / blurred / in- painted image.
This problem can be addressed using sparse representations whereby the de-noised image is generated from the noisy image as follows:
𝑧@ = 𝑦@ + 𝑥@, ∀𝑗 One observes a noisy version yi of image (patches) xi: The image (patches) xi obey a sparse representation zi in a dictionary D: 𝑦@ = 𝐸𝑨@, ∀𝑗 min
E,4F
G 𝑧@ − 𝐸𝑨@
I I @
+ 𝑨@
5
𝑦 J@ = 𝐸𝑨̂@, ∀𝑗
Low-resolution Image High-resolution Image
One then obtains the HR image from the LR image by determining the sparse representation associated with the images as well as the HR and LR dictionaries. One postulates that both the HR and the LR images admit a sparse representation in HR and LR dictionaries.
This problem can be addressed using sparse representations whereby the HR image is generated from the LR image as follows: One postulates that HR patches xiHR and LR patches xiLR admit a common sparse representation zi in HR and LR dictionaries DHR and DLR: 𝑦@
KL = 𝐸KL𝑨@, ∀𝑗
𝑦@
ML = 𝐸ML𝑨@, ∀𝑗
min
ENO,EPO,4F
G 𝑦@
KL − 𝐸KL𝑨@ I I + 𝑦@ ML − 𝐸ML𝑨@ I I + 𝜇 ⋅ 𝑨@ 5 @
𝑨̂@ = argmin
4F
𝑦@
ML − 𝐸ML𝑨@ I I + 𝜇 ⋅ 𝑨@ 5
𝑦 J@
KL = 𝐸KL𝑨̂@
Training: Testing:
Common Components
𝑦5 = ΦS 𝑨S + Φ 𝑨5 𝑦I = ΨS 𝑨S + Ψ 𝑨I
Innovation Components
data modality 1 data modality 2
Each individual image modalities admit sparse representations in a dictionary. The various image modalities are connected via sparse representations.
1. Model to represent accurately each individual image modality; 2. Model to connect the various image modalities; 3. Model to be readily learnt from data using simple algorithms; 4. Model to lead to simple multi-modal processing algorithms.
Our model can also be readily learnt using matrix factorization techniques.
T Samples dictionaries
Ψ 𝑌5 𝑌I 𝑎5 𝑎I 𝑎S
data matrix sparse matrix T Samples
Our model also leads to simple multi-modal image processing algorithms that exploit the joint sparse representations.
1. Model to represent accurately each individual image modality; 2. Model to connect the various image modalities; 3. Model to be readily learnt from data using simple algorithms; 4. Model to lead to simple multi-modal processing algorithms.
1. Model to represent accurately each individual image modality; 2. Model to connect the various image modalities; 3. Model to be readily learnt from data using simple algorithms; 4. Model to lead to simple multi-modal processing algorithms.
min
VW,V, XW,X YW,YZ,Y[
𝑌5 − ΦS 𝑎S − Φ𝑎5
\ I + 𝑌I − ΨS 𝑎S − Ψ𝑎I \ I
≤ 𝑡S, i = 1, …, 𝑈 𝑑𝑏𝑠𝑒 𝑎5 𝑗 ≤ 𝑡5,i = 1, …, 𝑈 𝑑𝑏𝑠𝑒 𝑎I 𝑗 ≤ 𝑡I,i = 1,… , 𝑈
Learn dictionaries by alternatingbetween:
(sparse coding step)
(dictionary update step)
Mixed X-Ray
Visual Front Panel Visual Rear Panel This problem involves separating the super-position
𝑧 = ΨS 𝑨 𝑦 = ΦS 𝑨 + Φ𝑤
Visual X-Ray
Visible X-Ray
The goal is to learn the joint parsimonious model from available data.
mixed x-ray visual front visual back
The goal is to unmix the x-rays given the x-ray mixture and the visuals.
mixed x-ray visuals in grayscale Ours multiscale MCA w/KSVD MCA reconstructed x-rays
Visual X-Rays Crack Mask
Mixed X-Rays Separation based on CDL Separation based on Weighted CDL
coupling between HR and LR image
This problem involves producing a high-resolution (HR) image from a low-resolution (LR) one of the same scene, by leveraging the presence of other images associated with the scene.
LR Image HR Image HR Side Information
HR image of interest LR image of interest another HR image
𝑧hi = ΦS
hi 𝑨S + Φhi𝑤
𝑦hi = ΨS
hi 𝑨S + Ψhi 𝑣
𝑦ki = ΨS
ki 𝑨S + Ψki 𝑣
This problem involves producing a high-resolution (HR) image from a low-resolution (LR) one of the same scene, by leveraging the presence of other images associated with the scene.
LR Image HR Image HR Side Information
HR image of interest LR image of interest another HR image
coupling between modalities 𝑧hi = ΦS
hi 𝑨S + Φhi𝑤
𝑦hi = ΨS
hi 𝑨S + Ψhi 𝑣
𝑦ki = ΨS
ki 𝑨S + Ψki 𝑣
This problem involves producing a high-resolution (HR) image from a low-resolution (LR) one of the same scene, by leveraging the presence of other images associated with the scene.
LR Image HR Image HR Side Information
HR image of interest LR image of interest another HR image
coupling between modalities 𝑧hi = ΦS
hi 𝑨S + Φhi𝑤
𝑦hi = ΨS
hi 𝑨S + Ψhi 𝑣
𝑦ki = ΨS
ki 𝑨S + Ψki 𝑣
Super-resolving hyper-spectral images with the aid of RGB images
LR-Image Ground Truth HR-Image - Bicubic HR-Image – Zeyde et al. HR-Image – A+ HR-Image – Ours Error - Bicubic Error – Zeyde et al. Error – A+ Error – Ours
Super-resolving infrared images with the aid of RGB images
LR-Image Ground Truth HR-Image - Bicubic HR-Image – Zeyde et al. HR-Image – A+ HR-Image – Ours Error - Bicubic Error – Zeyde et al. Error – A+ Error – Ours