Introduction to Computer Graphics Image Processing (1) July 4, - - PowerPoint PPT Presentation

Introduction to Computer Graphics – Image Processing (1) –

July 4, 2019 Kenshi Takayama

Today’s topics

• Edge-aware image processing
• Gradient-domain image processing

2

Image smoothing using Gaussian Filter

• Smoothness parameter 𝜏

3

𝜏 = 2 𝜏 = 5 𝜏 = 10 Original

Equation of Gaussian Filter

• 𝐽𝐪 represents pixel value of image 𝐽 at position 𝐪 = 𝑞x, 𝑞y ∈ Ω
• For given resolution e.g. 640×480, Ω ≔ 1, ⋯ , 640 × 1, ⋯ , 480
• GF𝜏[𝐽] represents filtered image with Gaussian parameter 𝜏 :
• 𝐻𝜏 𝑦 ≔ exp −

𝑦2 2𝜏2  Gaussian Kernel of radius 𝜏

4

𝑦 −𝜏 −2𝜏 −3𝜏 3𝜏 2𝜏 𝜏 𝐻𝜏 𝑦

GF𝜏 𝐽 𝐪 ≔ σ𝐫∈Ω 𝐻𝜏 𝐪 − 𝐫 𝐽𝐫 σ𝐫∈Ω 𝐻𝜏 𝐪 − 𝐫 𝑋

𝐪

Equation of Gaussian Filter

• 𝐽𝐪 represents pixel value of image 𝐽 at position 𝐪 = 𝑞x, 𝑞y ∈ Ω
• For given resolution e.g. 640×480, Ω ≔ 1, ⋯ , 640 × 1, ⋯ , 480
• GF𝜏[𝐽] represents filtered image with Gaussian parameter 𝜏 :
• 𝐻𝜏 𝑦 ≔ exp −

𝑦2 2𝜏2  Gaussian Kernel of radius 𝜏

5

𝑦 −𝜏 −2𝜏 −3𝜏 3𝜏 2𝜏 𝜏 𝐻𝜏 𝑦

GF𝜏 𝐽 𝐪 ≔ 1 𝑋

𝐪

෍

𝐫∈Ω

𝐻𝜏 𝐪 − 𝐫 𝐽𝐫

Implementing Gaussian Filter

• 𝐻𝜏 3𝜏 ≈ 0  Distant pixels can be ignored
• For fixed size 𝑠 ≔ ceil 3𝜏 , precompute weights
• n a (2𝑠 + 1) × (2𝑠 + 1) stencil

6

𝑦 −𝜏 −2𝜏 −3𝜏 3𝜏 2𝜏 𝜏 𝐻𝜏 𝑦

http://people.csail.mit.edu/sparis/bf_course/

2𝑠 + 1 2𝑠 + 1

Stencil

When kernel radius 𝜏 is very large

• Direct computation takes a lot of time
• Alternative: downsample  smooth with small 𝜏  upsample

7

𝜏 = 20 𝜏 = 5 downsample upsample

Detail Extraction & Enhancement

8

3 × smoothed detail enhanced detail halos!

When using edge-aware smoothing, ...

9

3 × smoothed detail enhanced detail

Edge-aware smoothing using Bilateral Filter

• Two parameters
• 𝜏s : Range of smoothing w.r.t. pixel’s location
• 𝜏r : Range of smoothing w.r.t. pixel’s color

BF𝜏s, 𝜏r 𝐽 𝐪 ≔ 1 𝑋

𝐪

෍

𝐫∈Ω

𝐻𝜏s 𝐪 − 𝐫 𝐻𝜏r 𝐽𝐪 − 𝐽𝐫 𝐽𝐫

10

𝜏r = 32 𝜏r = 128 𝜏r = 512 Original In all cases, 𝜏s = 10

Application of Bilateral Filter: Stylization

11

Real-time video abstraction [Winnemöller SIGGRAPH06]

Application of Bilateral Filter: Tone Mapping

• Range of each channel (24bit color image): 1~255
• Range of light intensity in the real world: 1~105
• High Dynamic Range image
• Can be obtained by photographing with different exposure times

12

https://en.wikipedia.org/wiki/Tone_mapping Fast bilateral filtering for the display of high-dynamic-range images [Durand SIGGRAPH02]

Short exposure Long exposure

Application of Bilateral Filter: Tone Mapping

13

γ correct. (𝑌 → 𝑌𝛿) details lost HDR image Bilateral Filter details preserved

https://en.wikipedia.org/wiki/Tone_mapping Fast bilateral filtering for the display of high-dynamic-range images [Durand SIGGRAPH02]

Naïve implementation of Bilateral Filter

1 𝑋

𝐪

෍

𝐫∈Ω

𝐻𝜏s 𝐪 − 𝐫 𝐻𝜏r 𝐽𝐪 − 𝐽𝐫 𝐽𝐫

• Recompute stencil for every

pixel location 𝐪 ∈ Ω  slow

• (Basic Assignment)

14

Another view toward Bilateral Filter

• Define feature vector 𝐠𝐪 ≔

𝐪 𝜏s , 𝐽𝐪 𝜏r for pixel location 𝐪 and intensity 𝐽𝐪

• Weight of Bilateral Filter is equivalent to

Gaussian kernel applied to Euclidean distance in the feature space

• Bilateral Filter is equivalent to applying Gaussian Filter of radius 1

to sample points {𝐠𝐪} in the feature space  Simpler computation

15

𝐻𝜏s 𝐪 − 𝐫 𝐻𝜏r 𝐽𝐪 − 𝐽𝐫 = exp − 𝐪 − 𝐫 2 2𝜏s

2

exp − 𝐽𝐪 − 𝐽𝐫

2

2𝜏r

2

= exp − 𝐠𝐪 − 𝐠𝐫

2

2 = 𝐻1 𝐠𝐪 − 𝐠𝐫

Bilateral Grid [Paris06; Chen07]

• Define 3D feature space as (X-coord, Y-coord, intensity),

map sample points {𝐠𝐪} to 3D grid

• The larger 𝜏s & 𝜏r, the coarser the grid  lower comput. cost

16

A Fast Approximation of the Bilateral Filter using a Signal Processing Approach [Paris ECCV06] Real-time edge-aware image processing with the bilateral grid [Chen SIGGRAPH07]

Weight map generation using feature space

• Various names: Edit Propagation, Matting, Segmentation
• Solve Laplace equation
• n Bilateral Grid

17

White scribble  constraint of weight=1 Black scribble  constraint of weight=0 Weight map Application: color adjustment

Real-time edge-aware image processing with the bilateral grid [Chen SIGGRAPH07]

Weight map generation using feature space

18

Instant Propagation of Sparse Edits on Images and Videos [Li PG10] Bilateral Hermite Radial Basis Functions for Contour-based Volume Segmentation [Ijiri EG13]

https://www.youtube.com/watch?v=mL6ig_OaQAA

Interpolation using RBF [Li10] (Purpose: edit propagation for images/videos) Interpolation using Hermite RBF [Ijiri13] (Purpose: segmentation of CT volume)

Extension of Bilateral Filter: Joint (Cross) Bilateral Filter

19

Digital Photography with Flash and No-Flash Image Pairs [Petschnigg SIGGRAPH04] Flash Photography Enhancement via Intrinsic Relighting [Eisemann SIGGRAPH04]

Photo A: without flash  Correct color  Noisy, blurred Photo F: with flash  Incorrect color  Less noisy, sharp

JBF𝜏s, 𝜏r 𝐵, 𝐺 𝐪 ≔ 1 𝑋

𝐪

෍

𝐫∈Ω

𝐻𝜏s 𝐪 − 𝐫 𝐻𝜏r 𝐺

𝐪 − 𝐺 𝐫

𝐵𝐫

After applying JBF

Extension of Bilateral Filter: Non-Local Means Filter

• Define feature space by neighborhood vector 𝐨𝐪

representing 7×7 sub-image centered at 𝐪

20

A non local algorithm for image denoising [Buades CVPR05]

NLMF𝜏 𝐽 𝐪 ≔ 1 𝑋

𝐪

෍

𝐫∈Ω

𝐻𝜏 𝐨𝐪 − 𝐨𝐫 𝐽𝐫

Noisy input Bilateral NL Means

Today’s topics

• Edge-aware image processing
• Gradient-domain image processing

21

Scenario: insert source img. into destination img.

22

Poisson image editing [Perez SIGGRAPH03] Gradient Domain Manipulation Techniques in Vision and Graphics [Agrawal ICCV07 Course]

Simple copying  Boundary blurred  Gradient-domain processing  Source Dest.

Scenario: generating panorama from several shots

23

Efficient gradient-domain compositing using quadtrees [Agarwala SIGGRAPH07]

Simple tiling Gradient-domain processing

Simple case of 1D grayscale image

24

Offset Dest. Source

2D case: offset by Laplace Membrane

25

Mean value coordinates [Floater CAGD03] Coordinates for instant image cloning [Farbman SIGGRAPH09]

• Solve Laplace equation under

Dirichlet boundary condition

• Fast approximation using

Mean Value Coordinates

https://www.youtube.com/watch?v=AXvPeuc-wRw

Gradient-domain processing in general form (not just simple cloning)

26

27

×2

||

Modify gradients arbitrarily!

Gradient-domain processing in general form (not just simple cloning)

28

𝑔∗ 𝑕𝑗 𝑔∗

Find 𝑔

𝑗

that minimize ෍

𝑗

𝑔

𝑗 − 𝑔 𝑗−1 − 𝑕𝑗 2

subject to: 𝑔ȁ𝜖Ω = 𝑔∗ȁ𝜖Ω

Ω

Find function 𝑔 such that its gradient closely matches user-specified target 𝑕𝑗

Gradient-domain processing in general form (not just simple cloning)

• Basics of Gradient-domain image processing:

Find image 𝑔 whose gradient best matches user-specified target gradient field 𝐡 by solving Poisson equation

29

Find 𝑔

𝑗

that minimize ෍

𝑗

𝑔

𝑗 − 𝑔 𝑗−1 − 𝑕𝑗 2

subject to: 𝑔ȁ𝜖Ω = 𝑔∗ȁ𝜖Ω Find 𝑔 𝑦, 𝑧 that minimizes න

𝑦,𝑧 ∈Ω

𝛂𝑔 𝑦, 𝑧 − 𝐡(𝑦, 𝑧) 2 subject to: 𝑔ȁ𝜖Ω = 𝑔∗ȁ𝜖Ω

1D case 2D case

Solve Poisson equation: Δ𝑔 = 𝛂 ⋅ 𝐡 subject to: 𝑔ȁ𝜖Ω = 𝑔∗ȁ𝜖Ω

⇔

How to give target gradients: mixing

• Copy source’s gradient only when its magnitude is larger

 smooth part of source won’t be copied

30

Poisson image editing [Perez SIGGRAPH03]

source dest. source dest. source dest.

How to give target gradient: Edge Brush

• Copy gradients along object silhouette,

paste along brush stroke

• Real-time Poisson solver implemented on GPU

31

Real-time gradient-domain painting [McCann SIGGRAPH08] Code: http://graphics.cs.cmu.edu/projects/gradient-paint/gradient_paint.r2403.tar.gz

https://www.youtube.com/watch?v=9MGjrsPzFc4

Before After Before After

How to give target gradient: modify original

32

Poisson image editing [Perez SIGGRAPH03]

Amplify/suppress within selected region  Local Tone Mapping Set to zero except where detected as edges  Stylization

Extra: Gradient-domain geometry processing

33

Gradient-domain geometry processing

34

Find 𝐰𝑗 that minimize ෍

𝑗,𝑘 ∈𝐹

𝑥𝑗𝑘 𝐰𝑗 − 𝐰

𝑘 − 𝐟𝑗𝑘 2

subject to: 𝐰𝑑 = 𝐰𝑑

∗, 𝑑 ∈ 𝐽𝐷

Mesh editing with poisson-based gradient field manipulation [Yu SIGGRAPH04] Laplacian surface editing [Sorkine SGP04] Interfaces and algorithms for the creation, modification, and optimization of surface meshes [Nealen PhD07]

Edge vector of original shape  target gradient A few constraints of vertex positions  boundary condition

Poisson equation

Rotation of local region due to large deformation

• Target gradient needs to be rotated as well
• Non-linear relation
• Optimal rotation difficult to find
• Local-global optimization [Sorkine07]
• Local step:

Fix vertex positions, compute local rotation using SVD

• Global step:

Fix local rotations, compute vertex positions via Poisson equation

35

On linear variational surface deformation methods [Botsch TVCG08] As-rigid-as-possible surface modeling [Sorkine SGP07]

https://www.youtube.com/watch?v=ltX-qUjbkdc

Without rotation With rotation

GeoBrush: Cloning brush for surface meshes

• Split deformation into two steps:
• 1. Rotation of local region

 Fast & approx. computation using cage-based method

• 2. Accurate offset

 Adapt GPU-based Poisson solver (originally for image processing)

36

GeoBrush: Interactive Mesh Geometry Cloning [Takayama EG11]

https://www.youtube.com/watch?v=FPsccn_gG8E