CFA Interpolation Detection Leszek Swirski October 15, 2009 - - PowerPoint PPT Presentation

▶
cfa interpolation detection
SMART_READER_LITE
LIVE PREVIEW

CFA Interpolation Detection Leszek Swirski October 15, 2009 - - PowerPoint PPT Presentation

Sep 29, 2023 163 likes •832 views

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis CFA Interpolation Detection Leszek Swirski October 15, 2009 Leszek Swirski CFA Interpolation Detection Outline CFA Introduction Interpolation Detection CFA

slide-1
SLIDE 1

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis

CFA Interpolation Detection

Leszek ´ Swirski October 15, 2009

Leszek ´ Swirski CFA Interpolation Detection

slide-2
SLIDE 2

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis

CFA Introduction Example Interpolation Interpolation Detection Methods Examples Identifying Forged Regions CFA Pattern Synthesis Reasoning Methods

Leszek ´ Swirski CFA Interpolation Detection

slide-3
SLIDE 3

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Example Interpolation

CFA?

▶ “Colour Filter Array” ▶ Photosensors have no

wavelength specificity

▶ So filter RGB onto array

  • f photosensors

▶ e.g. Bayer filter

Leszek ´ Swirski CFA Interpolation Detection

slide-4
SLIDE 4

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Example Interpolation

Target Image

Leszek ´ Swirski CFA Interpolation Detection

slide-5
SLIDE 5

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Example Interpolation

Filters

Leszek ´ Swirski CFA Interpolation Detection

slide-6
SLIDE 6

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Example Interpolation

Sensor Data

Leszek ´ Swirski CFA Interpolation Detection

slide-7
SLIDE 7

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Example Interpolation

Coloured Sensor Data

Leszek ´ Swirski CFA Interpolation Detection

slide-8
SLIDE 8

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Example Interpolation

Coloured Sensor Data (Detail)

Leszek ´ Swirski CFA Interpolation Detection

slide-9
SLIDE 9

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Example Interpolation

Why interpolate?

▶ Each pixel is only R, G or B ▶ Want full colour, full size image ▶ So guess interpolate!

Leszek ´ Swirski CFA Interpolation Detection

slide-10
SLIDE 10

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Example Interpolation

Nearest Neighbour

Leszek ´ Swirski CFA Interpolation Detection

slide-11
SLIDE 11

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Example Interpolation

Nearest Neighbour

Leszek ´ Swirski CFA Interpolation Detection

slide-12
SLIDE 12

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Example Interpolation

Nearest Neighbour

Leszek ´ Swirski CFA Interpolation Detection

slide-13
SLIDE 13

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Example Interpolation

Bilinear (and other polynomials)

R and B R′/B′ = R/B ∗ 1 4 ⎡ ⎣ 1 2 1 2 4 2 1 2 1 ⎤ ⎦ G G ′ = G ∗ 1 4 ⎡ ⎣ 1 1 4 1 1 ⎤ ⎦

Leszek ´ Swirski CFA Interpolation Detection

slide-14
SLIDE 14

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Example Interpolation

Bilinear (and other polynomials)

Leszek ´ Swirski CFA Interpolation Detection

slide-15
SLIDE 15

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Example Interpolation

Bilinear (and other polynomials)

Leszek ´ Swirski CFA Interpolation Detection

slide-16
SLIDE 16

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Example Interpolation

Smooth Hue Transition

Seperate luminance (G) and chrominance (R and B). Interpolate G bilinearly G ′ = G ∗ 1 4 ⎡ ⎣ 1 1 4 1 1 ⎤ ⎦ For R (and similarly for B), interpolate the ratio R′′

ij = Rij G ′

ij , and

pointwise multiply by G R′

ij = Gij ×

⎛ ⎝R′′ ∗ 1 4 ⎡ ⎣ 1 2 1 2 4 2 1 2 1 ⎤ ⎦ ⎞ ⎠

ij

Leszek ´ Swirski CFA Interpolation Detection

slide-17
SLIDE 17

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Example Interpolation

Smooth Hue Transition

Leszek ´ Swirski CFA Interpolation Detection

slide-18
SLIDE 18

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Example Interpolation

Smooth Hue Transition

Leszek ´ Swirski CFA Interpolation Detection

slide-19
SLIDE 19

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Example Interpolation

Median filter

▶ Bilinearly filter R, G, B to get R′′, G ′′, B′′. ▶ Calculate pairwise differences (R′′ − G ′′, R′′ − B′′, G ′′ − B′′) ▶ Median filter these to get Mrg, Mrb, Mgb ▶ Each resulting pixel is CFA pixel image plus/minus appropriate

median. e.g. (1,0) is a green pixel in CFA, so R′

1,0 = G1,0 + (Mrg)1,0

G ′

1,0 = G1,0

B′

1,0 = G0,0 − (Mgb)1,0

Leszek ´ Swirski CFA Interpolation Detection

slide-20
SLIDE 20

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Example Interpolation

Median filter

Leszek ´ Swirski CFA Interpolation Detection

slide-21
SLIDE 21

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Example Interpolation

Median filter

Leszek ´ Swirski CFA Interpolation Detection

slide-22
SLIDE 22

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Example Interpolation

Gradient-Based

▶ Want to preserve edges, so ‘adaptively’ interpolate G ▶ Approximate horizontal and vertical second derivatives of R

and B and take absolutes, e.g.

  • ∂2R

∂x2

  • i,j

≈

  • Ri,j−2 + Ri,j+2

2 − Ri,j

  • ▶ Compare these H and V . If Hi,j < Vi,j, (i,j) is a horizontal

edge, so interpolate horizontally. G ′

i,j =

⎧  ⎨  ⎩

Gi,j−1+Gi,j+1 2

Hi,j < V i, j

Gi−1,j+Gi+1,j 2

Hi,j > V i, j

Gi,j−1+Gi,j+1+Gi−1,j+Gi+1,j 4

Hi,j = V i, j

Leszek ´ Swirski CFA Interpolation Detection

slide-23
SLIDE 23

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Example Interpolation

Gradient-Based

Leszek ´ Swirski CFA Interpolation Detection

slide-24
SLIDE 24

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Example Interpolation

Gradient-Based

Leszek ´ Swirski CFA Interpolation Detection

slide-25
SLIDE 25

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Example Interpolation

And more...

▶ Adaptive Colour Plane

Has adaptive interpolation for G using first order derivative of luminance and second order derivative of chrominance, and uses adaptive interpolation again for R and B.

▶ Threshold-Based Variable Number of Gradients

Eight gradient samples taken from a 5 × 5 neighbourhood of each pixel, averages are calculated for each gradient, gradients

  • f values less than a (dynamic!) threshold are averaged, and

averages are added to/subtracted from the CFA values. (Though both are similar in principle to the gradient-based)

Leszek ´ Swirski CFA Interpolation Detection

slide-26
SLIDE 26

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Methods Examples Identifying Forged Regions

Why detect?

▶ Image/Camera verification ▶ Identification of forged regions ▶ Recognition of PRCG (PhotoRealistic Computer Generated

images)

Leszek ´ Swirski CFA Interpolation Detection

slide-27
SLIDE 27

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Methods Examples Identifying Forged Regions

General idea

▶ Interpolation creates correlation ▶ Most CFA interpolation is regular and approximately linear

(especially for G)

▶ If can determine some regular correlation, image is

interpolated

Leszek ´ Swirski CFA Interpolation Detection

slide-28
SLIDE 28

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Methods Examples Identifying Forged Regions

EM Algorithm

▶ ‘Expectation-Maximisation’ ▶ Simulataneously estimate parameters of correlation (i.e. what

interpolation is used) and which points are correlated to their neighbours

▶ Two-step iterative algorithm ▶ Creates a separable parameter space

Leszek ´ Swirski CFA Interpolation Detection

slide-29
SLIDE 29

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Methods Examples Identifying Forged Regions

A statistical Achilles’ Heel

▶ Interpolation creates correlation ▶ Correlation decreases variance ▶ Variance can be measured! ▶ Periodic low variance is indicative of interpolation

Leszek ´ Swirski CFA Interpolation Detection

slide-30
SLIDE 30

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Methods Examples Identifying Forged Regions

A statistical Achilles’ Heel

휎2

휎2 4

휎2

휎2 4

휎2

휎2 4

휎2

휎2 4

휎2

휎2 4

휎2

휎2 4

휎2

휎2 4

휎2

휎2 4

휎2

휎2 4

휎2

휎2 4

휎2

휎2 4

휎2

휎2 4

휎2

휎2 4

휎2

휎2 4

휎2

휎2 4

휎2

휎2 4

휎2

휎2 4

휎2

휎2 4

휎2

휎2 4

휎2

휎2 4

휎2

휎2 4

휎2

휎2 4

휎2

휎2 4

휎2

휎2 4

휎2

Leszek ´ Swirski CFA Interpolation Detection

slide-31
SLIDE 31

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Methods Examples Identifying Forged Regions

Gallagher and Chen

▶ First, high-pass filter:

h = ⎡ ⎣ 1 1 4 1 1 ⎤ ⎦

▶ Estimate variance using mean of absolutes along

anti-diagonals m(d) = ∑

x+y=d

∣(h ∗ i)x,y∣ Nd

Leszek ´ Swirski CFA Interpolation Detection

slide-32
SLIDE 32

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Methods Examples Identifying Forged Regions

Gallagher and Chen

▶ Want to find periodicity, so DFT to get

  • M(ei휔)
  • ▶ Significant peaks demonstrate periodicity

▶ Green channel interpolates every other pixel, so expect a peak

at 휔 = 휋

▶ Quantify peak s as:

s =

  • M(ei휔)
  • 휔=휋

median휔{∣M(ei휔)∣}

Leszek ´ Swirski CFA Interpolation Detection

slide-33
SLIDE 33

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Methods Examples Identifying Forged Regions

Gallagher and Chen example 1

Original

Leszek ´ Swirski CFA Interpolation Detection

slide-34
SLIDE 34

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Methods Examples Identifying Forged Regions

Gallagher and Chen example 1

i =

Leszek ´ Swirski CFA Interpolation Detection

slide-35
SLIDE 35

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Methods Examples Identifying Forged Regions

Gallagher and Chen example 1

(h ∗ i) =

Leszek ´ Swirski CFA Interpolation Detection

slide-36
SLIDE 36

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Methods Examples Identifying Forged Regions

Gallagher and Chen example 1

m(d) = 0.2 0.4 0.6 0.8 1 5 ⋅ 10−2 0.1

Leszek ´ Swirski CFA Interpolation Detection

slide-37
SLIDE 37

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Methods Examples Identifying Forged Regions

Gallagher and Chen example 1

  • M(ei휔)
  • =

휋 2

휋

3휋 2

2휋 50 100 150

Leszek ´ Swirski CFA Interpolation Detection

slide-38
SLIDE 38

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Methods Examples Identifying Forged Regions

Gallagher and Chen example 1

s =

  • M(ei휔)
  • 휔=휋

median휔{∣M(ei휔)∣} = 79.2337 0.1191 = 665.0172

Leszek ´ Swirski CFA Interpolation Detection

slide-39
SLIDE 39

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Methods Examples Identifying Forged Regions

Gallagher and Chen example 2

Original

Leszek ´ Swirski CFA Interpolation Detection

slide-40
SLIDE 40

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Methods Examples Identifying Forged Regions

Gallagher and Chen example 2

i =

Leszek ´ Swirski CFA Interpolation Detection

slide-41
SLIDE 41

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Methods Examples Identifying Forged Regions

Gallagher and Chen example 2

(h ∗ i) =

Leszek ´ Swirski CFA Interpolation Detection

slide-42
SLIDE 42

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Methods Examples Identifying Forged Regions

Gallagher and Chen example 2

m(d) = 0.2 0.4 0.6 0.8 1 5 ⋅ 10−2 0.1 0.15 0.2 0.25

Leszek ´ Swirski CFA Interpolation Detection

slide-43
SLIDE 43

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Methods Examples Identifying Forged Regions

Gallagher and Chen example 2

  • M(ei휔)
  • =

휋 2

휋

3휋 2

2휋 20 40 60 80 100

Leszek ´ Swirski CFA Interpolation Detection

slide-44
SLIDE 44

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Methods Examples Identifying Forged Regions

Gallagher and Chen example 2

s =

  • M(ei휔)
  • 휔=휋

median휔{∣M(ei휔)∣} = 0.2546 0.1688 = 1.5081

Leszek ´ Swirski CFA Interpolation Detection

slide-45
SLIDE 45

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Methods Examples Identifying Forged Regions

Gallagher and Chen example 3

Original

Leszek ´ Swirski CFA Interpolation Detection

slide-46
SLIDE 46

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Methods Examples Identifying Forged Regions

Gallagher and Chen example 3

i =

Leszek ´ Swirski CFA Interpolation Detection

slide-47
SLIDE 47

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Methods Examples Identifying Forged Regions

Gallagher and Chen example 3

(h ∗ i) =

Leszek ´ Swirski CFA Interpolation Detection

slide-48
SLIDE 48

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Methods Examples Identifying Forged Regions

Gallagher and Chen example 3

m(d) = 0.2 0.4 0.6 0.8 1 2 ⋅ 10−2 4 ⋅ 10−2 6 ⋅ 10−2 8 ⋅ 10−2

Leszek ´ Swirski CFA Interpolation Detection

slide-49
SLIDE 49

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Methods Examples Identifying Forged Regions

Gallagher and Chen example 3

  • M(ei휔)
  • =

휋 2

휋

3휋 2

2휋 2 4 6 8 10 12

Leszek ´ Swirski CFA Interpolation Detection

slide-50
SLIDE 50

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Methods Examples Identifying Forged Regions

Gallagher and Chen example 3

s =

  • M(ei휔)
  • 휔=휋

median휔{∣M(ei휔)∣} = 6.1377 0.0574 = 106.8595

Leszek ´ Swirski CFA Interpolation Detection

slide-51
SLIDE 51

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Methods Examples Identifying Forged Regions

Gallagher and Chen example 4

Original

Leszek ´ Swirski CFA Interpolation Detection

slide-52
SLIDE 52

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Methods Examples Identifying Forged Regions

Gallagher and Chen example 4

i =

Leszek ´ Swirski CFA Interpolation Detection

slide-53
SLIDE 53

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Methods Examples Identifying Forged Regions

Gallagher and Chen example 4

(h ∗ i) =

Leszek ´ Swirski CFA Interpolation Detection

slide-54
SLIDE 54

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Methods Examples Identifying Forged Regions

Gallagher and Chen example 4

m(d) = 0.2 0.4 0.6 0.8 1 2 ⋅ 10−2 4 ⋅ 10−2 6 ⋅ 10−2

Leszek ´ Swirski CFA Interpolation Detection

slide-55
SLIDE 55

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Methods Examples Identifying Forged Regions

Gallagher and Chen example 4

  • M(ei휔)
  • =

휋 2

휋

3휋 2

2휋 5 10 15

Leszek ´ Swirski CFA Interpolation Detection

slide-56
SLIDE 56

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Methods Examples Identifying Forged Regions

Gallagher and Chen example 4

s =

  • M(ei휔)
  • 휔=휋

median휔{∣M(ei휔)∣} = 2.8713 0.0637 = 45.1039

Leszek ´ Swirski CFA Interpolation Detection

slide-57
SLIDE 57

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Methods Examples Identifying Forged Regions

Problems (that aren’t)

▶ Non-linear interpolation makes this less simple ▶ JPEG introduces compression ▶ The demonstrated method doesn’t give the interpolation

parameters

▶ Can’t you fake CFA interpolation?

Leszek ´ Swirski CFA Interpolation Detection

slide-58
SLIDE 58

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Methods Examples Identifying Forged Regions

Problems (that aren’t)

▶ Non-linear interpolation makes this less simple

But non-linear interpolation is still locally sufficiently linear, so it’s close enough

▶ JPEG introduces compression ▶ The demonstrated method doesn’t give the interpolation

parameters

▶ Can’t you fake CFA interpolation?

Leszek ´ Swirski CFA Interpolation Detection

slide-59
SLIDE 59

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Methods Examples Identifying Forged Regions

Problems (that aren’t)

▶ Non-linear interpolation makes this less simple

But non-linear interpolation is still locally sufficiently linear, so it’s close enough

▶ JPEG introduces compression

But high-quality JPEG leaves enough information

▶ The demonstrated method doesn’t give the interpolation

parameters

▶ Can’t you fake CFA interpolation?

Leszek ´ Swirski CFA Interpolation Detection

slide-60
SLIDE 60

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Methods Examples Identifying Forged Regions

Problems (that aren’t)

▶ Non-linear interpolation makes this less simple

But non-linear interpolation is still locally sufficiently linear, so it’s close enough

▶ JPEG introduces compression

But high-quality JPEG leaves enough information

▶ The demonstrated method doesn’t give the interpolation

parameters No, but we don’t need them just for detection

▶ Can’t you fake CFA interpolation?

Leszek ´ Swirski CFA Interpolation Detection

slide-61
SLIDE 61

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Methods Examples Identifying Forged Regions

Problems (that aren’t)

▶ Non-linear interpolation makes this less simple

But non-linear interpolation is still locally sufficiently linear, so it’s close enough

▶ JPEG introduces compression

But high-quality JPEG leaves enough information

▶ The demonstrated method doesn’t give the interpolation

parameters No, but we don’t need them just for detection

▶ Can’t you fake CFA interpolation?

Sure, but adding such a method of detection makes forgery harder

Leszek ´ Swirski CFA Interpolation Detection

slide-62
SLIDE 62

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Methods Examples Identifying Forged Regions

Identifying Forged Regions

▶ Can identify forged regions by running the above algorithm

locally on each pixel

▶ For each pixel, estimate local (within radius n) variance

m(x, y) =

n

∑

i=−n

∣(h ∗ i)x+i,y+i∣ 2n + 1

▶ For each pixel, calculate local DFT, and from it the peak sxy ▶ Low values of sxy means pixel (x, y) part of forged region

Leszek ´ Swirski CFA Interpolation Detection

slide-63
SLIDE 63

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Reasoning Methods

Tamper hiding

▶ Tampering with a photo destroys CFA correlations ▶ Restoring CFA-like correlations hides tampering ▶ Want an image of similar size and quality

Leszek ´ Swirski CFA Interpolation Detection

slide-64
SLIDE 64

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Reasoning Methods

Na¨ ıve method

▶ Can simply sample image into CFA image, and reinterpolate ▶ Problem: Throws away 2 3 of pixel data! ▶ Similar information loss to linear kernel smoothing (e.g.

Gaussian)

Leszek ´ Swirski CFA Interpolation Detection

slide-65
SLIDE 65

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Reasoning Methods

‘Ideal’ method

▶ Can represent linear interpolation as matrix operation

y = Hx

▶ Manipulated image adds an additive residual signal

y = Hx + 휖

▶ For given H, want to find x which minimises ∥휖∥

Leszek ´ Swirski CFA Interpolation Detection

slide-66
SLIDE 66

Outline CFA Introduction Interpolation Detection CFA Pattern Synthesis Reasoning Methods

‘Ideal’ method

▶ This is a least squares problem, with solution

x = (HTH)−1HT

  • Moore-Penrose Pseudoinverse

y

▶ Can therefore synthesise a least error CFA pattern

yCFA = H(HTH)−1HTy

▶ But calculating (HTH)−1HT efficiently is tricky...

Leszek ´ Swirski CFA Interpolation Detection