- An overview - Marcel Lthi Graphics and Vision Research Group - PowerPoint PPT Presentation

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL Probabilistic Morphable Models - An overview - Marcel Lüthi Graphics and Vision Research Group Department of Mathematics and Computer Science University of Basel

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL Outline Comparison Analysis by synthesis Parame Update 𝜄 Synthesis 𝜒(𝜄) ters 𝜄 Medical image analysis verse computer vision The space of images

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL Probabilistic Morphable Models Online Course Summer school Shape Modelling Model fitting Scalismo

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL Conceptual Basis: Analysis by synthesis Comparison Parameters 𝜄 Update 𝜄 Synthesis 𝜒(𝜄) • If we are able to synthesize an image, we can explain it. • We can explain unseen parts and reason about them

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL Synthesizing images Comparison Parameters 𝜄 Update 𝜄 Synthesis 𝜒(𝜄) Computer graphics

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL Synthesizing images Comparison Parameters 𝜄 Update 𝜄 Synthesis 𝜒(𝜄) • Warping atlas images • Simulating DRRs • Virtual CTs

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL Mathematical Framework: Bayesian inference Comparison: 𝑞 Image 𝜄) Prior 𝑞(𝜄) Parameters 𝜄 Update using 𝑞(𝜄|Image) Synthesis 𝜒(𝜄) • Principled way of dealing with uncertainty.

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL Algorithmic implementation: MCMC Comparison: 𝑞 Image 𝜄) Prior 𝑞(𝜄) Parameters 𝜄 Sample from 𝑞(𝜄|Image) Synthesis 𝜒(𝜄)

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL Course programme Mor orning Aft fternoon • Introduction • Basic tasks in Scalismo-faces Tuesday • The course at a glance • Ba Bayesian mod odel elli ling • Basics of Co Computer Graphics • Welcome reception • Probabilistic mod • Exercises: MCMC Fitting Wednesday odel l fit fitti ting • Introduction to course project usin ing g MCM CMC • Face im • Course project Thursday image analy lysis • Connections to medical image • Course project Friday analysis • Advanced topics in Gaussian processes • Project presentation • Social event Saturday

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL The course in context Pattern Theory Text Music Ulf Grenander Computational Research at anatomy Gravis Medical Images Fotos Speech This course / Natural language PMM 11

12 > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL Pattern theory vs PMM • Pattern theory is about developing a theory for understanding real- world signals • Probabilistic Morphable Models are about usin ing theoretical well founded concepts to analyse images. • GPs for modelling • MCMC for model fitting • Working software

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL Images: Medical Image Analysis vs Computer Vision Source: OneYoungWorld.com

18 > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL Images in medical image analysis Goal: Measure and visualize the unseen • Acquired with specific purpose • Controlled measurement • Done by experts • Calibrated, specialized devices Source: www.siemens.com

19 > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL Images in medical image analysis (0,0,0) • Images live in a coordinate system (units: mm)

20 > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL Images in medical image analysis 280 𝑛𝑛 300 𝑛𝑛 (100,720, 800) (0,0,0)

21 > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL Images in medical image analysis Values measure properties of the patient’s tissue • Usually scalar-valued • Often calibrated • CT Example: -1000 HU -> Air 3000 HU -> cortical bone

24 > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL Images in computer vision Goal: Capture what we see in a realistic way • Perspective projection from 3D object to 2D image • Many parts are occluded

25 > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL Images in computer vision • Can be done by anybody • Acquisition device usually unknown • Uncontrolled background, lighting, … • No clear scale • What is the camera distance? • No natural coordinate system • Unit usually pixel Source: twitter.com

26 > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL Images in computer vision • Pixels represent RGB values • Values are measurement of light • Reproduce what the human eye would see • Exact RGB value depends strongly on lighting conditions • Shadows • Ambient vs diffuse light

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL Images: Medical Image analysis vs Computer Vision Computer vis vision Medical l im image • Uncontrolled snapshot • Controlled measurement • Values are mixture of different • Values have (often) clear (unknown factors) interpretation • Many occlusion due to perspective • Explicit setup to visualize unseen • Scale unknown • Coordinate system with clear scale Many complications of computer vision arise in different form also in a medical setting.

28 > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL The space of images

29 > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL The space of images

30 > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL The space of images • 10 x 10 image • 2 colours How long would you need to watch TV (24 fps) until you have seen all such images? • Number of images: 2 100 ≈ 1.2e30 Source: bbc.co.uk • Watching 100 years continuously 24 x 60 x 60 x 24 x 365 x 100 ≈ 7.5e10

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL The space of images Think of interesting • Most images are uninteresting images like this • Only very few of all possible images are of interest to us Not this ℝ 100

32 > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL Structure in images • Images have a rich structure Our mission: • Images are depictions of ob objects in the • Model this structure world • • Structure is due to objects, laws and Needs only few parameters processes • Explain image by finding appropriate parameters that reflect objects / laws / processes

> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 | BASEL Next stop: Computer graphics Comparison: 𝑞 Image 𝜄) Image Parameters 𝜄 Sample from 𝑞(𝜄|Image) Synthesis 𝜒(𝜄) • Computer graphics