Sisley the Abstract Painter Mingtian Zhao Song-Chun Zhu University - - PowerPoint PPT Presentation

NPAR 2010

Sisley the Abstract Painter

Mingtian Zhao Song-Chun Zhu University of California, Los Angeles & Lotus Hill Institute

Motivation

Wheatstack (Thaw, Sunset) Claude Monet 1890–91 “I considered that the painter had no right to paint indistinctly . . . and I noticed with surprise and confusion that the picture not only gripped me, but impressed itself ineradicably on my memory.” — Wassily Kandinsky

Perceptual Ambiguity

• Indistinction, Confusion: Perceptual Ambiguity
• The Mechanism of Abstract Arts [Berlyne 1971]

Perceptual Ambiguity ⇒ Mental Efforts ⇒ Arousal Changes ⇒ Aesthetic Pleasures

• Where does perceptual ambiguity come from?

what we see = arg max P(interpretation|image)

More Abstract Arts

• No. 5, 1948

Jackson Pollock Violin and Guitar Pablo Picasso

More Abstract Arts

Le Mont Sainte-Victoire Paul C´ ezanne The Red Vineyard Vincent van Gogh

More Abstract Arts

• They all preserve certain features and free others.
• Features are marginal statistics.
• Projection Pursuit [Friedman & Tukey 1974]
• Semantic Fidelity vs. Uncertainty & Ambiguity
• Paths of Perception

Our Approach

• Interactive Image Parsing [Tu et al. 2005]

⋄ Interactive Segmentation ⋄ Hierarchical Organization ⋄ Category Labeling (optional but recommended)

• Customization and Rendering

⋄ Customized Perceptual Ambiguity Levels ⋄ Stochastic Operations on Color/Shape/Texture

• Computation and Control

⋄ Kernel Density Estimation ⋄ Belief Propagation ⋄ Servomechanism

Parse Tree

seascape sailboat sea buildings trees sky sail hull

Numerical Measure of Perceptual Ambiguity

• Assume parse tree structure is obvious, and
• Perceptual ambiguity is only with categories

L = (ℓ1, ℓ2, · · · , ℓK)

• Uncertainty in p(L): Information/Shannon Entropy

H(L)|I =

• L

−p(L|I) log p(L|I)

Stochastic Operations

• Color

⋄ hue shift ∆h ∼ GT(0, σ2, −3σ, 3σ) σ ∝ H, σmax = 15◦

• Shape

⋄ boundary pixel shift (∆x, ∆y ∼ GT) ⋄ image warping

• Texture

⋄ Painterly Rendering (adapted from [Zeng et al. 2009])

Computation & Control

• Parse Tree as Markov random field (MRF)

⋄ Hammersley-Clifford p(L|I) = 1 Z

• i∈V

φi(ℓi)

• i,j∈E

ψij(ℓi, ℓj) ⋄ Unary Term: Local Evidence φi(ℓi) ← p(ℓi|Ii) ⋄ Binary Term: Compatibility ψij(ℓi, ℓj) ← f(ℓi, ℓj)

Computation & Control

• Local Evidence by Kernel Density Estimation

Distant: low-weight votes Close: high-weight votes

• The LHI Image Database

Computation & Control

• Local Evidence by Kernel Density Estimation

p(ℓi|Ii) ∝

• n

ϕ(Ii, Jn)1(ℓi = ℓn)

• The Exponential Kernel

ϕ(Ii, Jn) = exp {−λs(Ii) − s(Jn)} s : Opponent-SIFT [van de Sande et al. 2010]

• The use of manually labeled categories
• Can we plug in p(ℓi|Ii) to get p(L|I) and H(L)|I?

Computation & Control

• Problem of computing H(L)|I:

The space of L is usually huge. |ΩL| = (#categories)#nodes

• Approximation with p(ℓi|I) instead of p(ℓi|Ii)

H(L)|I ≈

• i

wiH(ℓi)|I wi ∝ size(i) H(ℓi)|I =

• L

−p(ℓi|I) log p(ℓi|I)

Computation & Control

• Belief Propagation on MRF [Yedidia et al. 2001]

Ii i bi(ℓi) j k mij mji mik mki p(ℓi|Ii)

Computation & Control

• Belief Propagation on MRF [Yedidia et al. 2001]

⋄ Update Messages mij(ℓj) =

• ℓi

p(ℓi|Ii) f(ℓi, ℓj)

• k∈∂i\j

mki(ℓi) ⋄ Update Beliefs bi(ℓi) ∝ p(ℓi|Ii)

• j∈∂i

mji(ℓi) ⋄ p(ℓi|I) ← bi(ℓi) after convergence

Computation & Control

• Relative Abstract Level
• H =
• i wiH(ℓi)|I
• i wi log
• Ωℓi
• ∈ [0, 1]
• Servomechanism

⋄ If H(t)

• ut <

H(t)

in

• H(t+1)

I

= H(t)

in

2

• H(t)
• ut

⋄ If H(t)

• ut >

H(t)

in

• H(t+1)

in

= 1 −

• 1 −

H(t)

in

2 1 − H(t)

• ut

Experimental Results

• H ≈ 0
• H ≈ 0.25
• H ≈ 0.5
• H ≈ 0.75

Experimental Results

Experimental Results

Experimental Results

Different Paths of Perception

Evaluations

• Comparative Human Experiments

Photographs Original Paintings Synthesized Paintings Alley Flying Bird Buildings Butterfly

Evaluations

• Recognition Accuracy

Photographs Original Paintings Synthesized Paintings Horizontal axis: reported categories Vertical axis: true categories Darkness of grids: frequencies

Evaluations

• Response Speed

⋄ ANOVA F-test: p-value=2.955 × 10−8 ⋄ Tukey’s HSD

Group Pair ∆¯ t (ms) p-value Photographs vs. Original Paintings 2165 < 0.01 Photographs vs. Synthesized Paintings 1183 0.03 Original vs. Synthesized Paintings −982 0.11

Summary & Future Studies

• Abstract Arts Rendering by Perceptual Ambiguity

Computation and Control

• Convergence control
• Variable parse tree structures
• Mixture modeling of abstract arts

More Results and Demo System: http://www.stat.ucla.edu/∼mtzhao/research/sisley/