Model-Based Teeth Reconstruction Chenglei Wu 2 , Derek Bradley 1 , - - PowerPoint PPT Presentation

▶
model based teeth reconstruction
SMART_READER_LITE
LIVE PREVIEW

Model-Based Teeth Reconstruction Chenglei Wu 2 , Derek Bradley 1 , - - PowerPoint PPT Presentation

Feb 17, 2023 465 likes •1.48k views

Model-Based Teeth Reconstruction Chenglei Wu 2 , Derek Bradley 1 , Pablo Garrido 3 , Michael Zollhfer 3 , Christian Theobalt 3 , Markus Gross 1,2 , Thabo Beeler 1 1 Disney Research 2 ETH Zurich 3 Max Planck Institute for Informatics MOTIVATION

slide-1
SLIDE 1

Chenglei Wu2, Derek Bradley1, Pablo Garrido3, Michael Zollhöfer3, Christian Theobalt3, Markus Gross1,2, Thabo Beeler1

1Disney Research 2ETH Zurich 3Max Planck Institute for Informatics

Model-Based Teeth Reconstruction

slide-2
SLIDE 2

Michael Zollhöfer Model-Based Teeth Reconstruction 2

MOTIVATION

Michael Zollhöfer Model-Based Teeth Reconstruction 2

slide-3
SLIDE 3

Michael Zollhöfer Model-Based Teeth Reconstruction 3

MOTIVATION

Input Image Beeler11

Michael Zollhöfer Model-Based Teeth Reconstruction 3

slide-4
SLIDE 4

Michael Zollhöfer Model-Based Teeth Reconstruction 4

MOTIVATION

Input Image Beeler11

Michael Zollhöfer Model-Based Teeth Reconstruction 4

Berard16

slide-5
SLIDE 5

Michael Zollhöfer Model-Based Teeth Reconstruction 5

MOTIVATION

Input Image Beeler11

Michael Zollhöfer Model-Based Teeth Reconstruction 5

Berard16 Bermano16

slide-6
SLIDE 6

Michael Zollhöfer Model-Based Teeth Reconstruction 6

MOTIVATION

Input Image Beeler11

Michael Zollhöfer Model-Based Teeth Reconstruction 6

Beeler12 Berard16 Bermano16

slide-7
SLIDE 7

Michael Zollhöfer Model-Based Teeth Reconstruction 7

MOTIVATION

Input Image Beeler11

Michael Zollhöfer Model-Based Teeth Reconstruction 7

Beeler12 Berard16 Bermano16 Nagano15

slide-8
SLIDE 8

Michael Zollhöfer Model-Based Teeth Reconstruction 8

MOTIVATION

Input Image Beeler11

Michael Zollhöfer Model-Based Teeth Reconstruction 8

Beeler12 Berard16 Bermano16 Nagano15 Hu14

slide-9
SLIDE 9

Michael Zollhöfer Model-Based Teeth Reconstruction 9

Teeth are missing!

MOTIVATION

Input Image Beeler11

slide-10
SLIDE 10

Michael Zollhöfer Model-Based Teeth Reconstruction 10

MOTIVATION

Input Image Beeler11 Teeth are missing!

slide-11
SLIDE 11

Michael Zollhöfer Model-Based Teeth Reconstruction 11

Complex Appearance

MOTIVATION

Input Image Beeler11 Teeth are missing!

slide-12
SLIDE 12

Michael Zollhöfer Model-Based Teeth Reconstruction 12

Complex Appearance Featureless

MOTIVATION

Input Image Beeler11 Teeth are missing!

slide-13
SLIDE 13

Michael Zollhöfer Model-Based Teeth Reconstruction 13

Complex Appearance Occlusions Featureless

MOTIVATION

Input Image Beeler11 Teeth are missing!

slide-14
SLIDE 14

Michael Zollhöfer Model-Based Teeth Reconstruction 14

Complex Appearance Occlusions Featureless Input Image Our Reconstruction Teeth are missing!

MOTIVATION

slide-15
SLIDE 15

Michael Zollhöfer Model-Based Teeth Reconstruction 15

RELATED WORK

slide-16
SLIDE 16

Michael Zollhöfer Model-Based Teeth Reconstruction 16

Medical Dentistry

RELATED WORK

From CT images [Omachi07, Yanagisawa14]

slide-17
SLIDE 17

Michael Zollhöfer Model-Based Teeth Reconstruction 17

Medical Dentistry

RELATED WORK

From CT images [Omachi07, Yanagisawa14] Intra-Oral Scanners [3M True Definition, iTero, 3Shape TRIOS]

slide-18
SLIDE 18

Michael Zollhöfer Model-Based Teeth Reconstruction 18

Medical Dentistry

RELATED WORK

From CT images [Omachi07, Yanagisawa14] Intra-Oral Scanners [3M True Definition, iTero, 3Shape TRIOS]

Photogrammetric Reconstruction

Shape from Shading [Carter10, Farag13, Mostafa14]

slide-19
SLIDE 19

Michael Zollhöfer Model-Based Teeth Reconstruction 19

Medical Dentistry

RELATED WORK

From CT images [Omachi07, Yanagisawa14] Intra-Oral Scanners [3M True Definition, iTero, 3Shape TRIOS]

Photogrammetric Reconstruction

Shape from Shading [Carter10, Farag13, Mostafa14] Contours and Feature Points [Zheng11]

slide-20
SLIDE 20

Michael Zollhöfer Model-Based Teeth Reconstruction 20

Medical Dentistry

RELATED WORK

From CT images [Omachi07, Yanagisawa14] Intra-Oral Scanners [3M True Definition, iTero, 3Shape TRIOS]

Photogrammetric Reconstruction

Shape from Shading [Carter10, Farag13, Mostafa14] Contours and Feature Points [Zheng11] Statistical Models [Mehl05, Buchaillard07]

slide-21
SLIDE 21

Michael Zollhöfer Model-Based Teeth Reconstruction 21

HUMAN TEETH

Scans

slide-22
SLIDE 22

Michael Zollhöfer Model-Based Teeth Reconstruction 22

HUMAN TEETH

Scans

slide-23
SLIDE 23

Michael Zollhöfer Model-Based Teeth Reconstruction 23

HUMAN TEETH

Scans

slide-24
SLIDE 24

Michael Zollhöfer Model-Based Teeth Reconstruction 24

OVERVIEW

Scans

slide-25
SLIDE 25

Michael Zollhöfer Model-Based Teeth Reconstruction 25

OVERVIEW

Scans Static Images/Video Clip

slide-26
SLIDE 26

Michael Zollhöfer Model-Based Teeth Reconstruction 26

OVERVIEW

Scans Contour Extraction Static Images/Video Clip

slide-27
SLIDE 27

Michael Zollhöfer Model-Based Teeth Reconstruction 27

OVERVIEW

Teeth Prior Model

Scans Dental Scans Contour Extraction Static Images/Video Clip

slide-28
SLIDE 28

Michael Zollhöfer Model-Based Teeth Reconstruction 28

OVERVIEW

Teeth Prior Model

Scans

Teeth Reconstruction

Contour Extraction Dental Scans Static Images/Video Clip

slide-29
SLIDE 29

Michael Zollhöfer Model-Based Teeth Reconstruction 29

OVERVIEW

Teeth Prior Model 3D Reconstruction

Scans

Teeth Reconstruction

Contour Extraction Static Images/Video Clip Dental Scans

slide-30
SLIDE 30

Michael Zollhöfer Model-Based Teeth Reconstruction 30

OVERVIEW

Teeth Prior Model 3D Reconstruction

Scans

Teeth Reconstruction

Contour Extraction Dental Scans Static Images/Video Clip

slide-31
SLIDE 31

Michael Zollhöfer Model-Based Teeth Reconstruction 31

OVERVIEW

Teeth Prior Model 3D Reconstruction

Scans

Teeth Reconstruction

Contour Extraction Dental Scans Static Images/Video Clip

slide-32
SLIDE 32

Michael Zollhöfer Model-Based Teeth Reconstruction 32

TEETH MODEL

  • 86 high resolution plaster cast scans
slide-33
SLIDE 33

Michael Zollhöfer Model-Based Teeth Reconstruction 33

TEETH MODEL

  • 86 high resolution plaster cast scans
slide-34
SLIDE 34

Michael Zollhöfer Model-Based Teeth Reconstruction 34

Teeth Segmentation Fitted Template

PREPROCESSING

slide-35
SLIDE 35

Michael Zollhöfer Model-Based Teeth Reconstruction 35

Input Scan Teeth Segmentation Fitted Template

PREPROCESSING

slide-36
SLIDE 36

Michael Zollhöfer Model-Based Teeth Reconstruction 36

Input Scan Teeth Segmentation Fitted Template

PREPROCESSING

slide-37
SLIDE 37

Michael Zollhöfer Model-Based Teeth Reconstruction 37

Input Scan Teeth Segmentation Fitted Template

PREPROCESSING

slide-38
SLIDE 38

Michael Zollhöfer Model-Based Teeth Reconstruction 38

𝐚𝛖 = 𝐓 𝑼τ 𝐁𝛖 +

𝐣

𝛃𝛖

𝐣 𝐂𝛖 𝐣

PARAMETRIC TEETH MODEL

slide-39
SLIDE 39

Michael Zollhöfer Model-Based Teeth Reconstruction 39

𝐚𝛖 = 𝐓 𝑼τ 𝐁𝛖 +

𝐣

𝛃𝛖

𝐣 𝐂𝛖 𝐣

PARAMETRIC TEETH MODEL

𝐁𝛖

slide-40
SLIDE 40

Michael Zollhöfer Model-Based Teeth Reconstruction 40

𝐚𝛖 = 𝐓 𝑼τ 𝐁𝛖 +

𝐣

𝛃𝛖

𝐣 𝐂𝛖 𝐣

PARAMETRIC TEETH MODEL

𝛃𝛖

𝐣 𝐂𝛖 𝐣

𝐁𝛖

shape of tooth 𝛖

slide-41
SLIDE 41

Michael Zollhöfer Model-Based Teeth Reconstruction 41

𝐚𝛖 = 𝐓 𝑼τ 𝐁𝛖 +

𝐣

𝛃𝛖

𝐣 𝐂𝛖 𝐣

PARAMETRIC TEETH MODEL

𝛃𝛖

𝐣 𝐂𝛖 𝐣

𝑼τ 𝐁𝛖

position of tooth 𝛖 shape of tooth 𝛖

slide-42
SLIDE 42

Michael Zollhöfer Model-Based Teeth Reconstruction 42

𝐚𝛖 = 𝐓 𝑼τ 𝐁𝛖 +

𝐣

𝛃𝛖

𝐣 𝐂𝛖 𝐣

PARAMETRIC TEETH MODEL

𝛃𝛖

𝐣 𝐂𝛖 𝐣

𝑼τ 𝐁𝛖 𝐓

Scale and position position of tooth 𝛖 shape of tooth 𝛖

slide-43
SLIDE 43

Michael Zollhöfer Model-Based Teeth Reconstruction 43

GAUSSIAN DISTRIBUTION

𝒀 = (𝐓 , 𝑼τ, {𝛃𝛖

𝐣 })

slide-44
SLIDE 44

Michael Zollhöfer Model-Based Teeth Reconstruction 44

GAUSSIAN DISTRIBUTION

  • All dimensions modeled via multivariate Gaussians

𝒀 = (𝐓 , 𝑼τ, {𝛃𝛖

𝐣 })

slide-45
SLIDE 45

Michael Zollhöfer Model-Based Teeth Reconstruction 45

GAUSSIAN DISTRIBUTION

  • All dimensions modeled via multivariate Gaussians

𝒀 = (𝐓 , 𝑼τ, {𝛃𝛖

𝐣 })

slide-46
SLIDE 46

Michael Zollhöfer Model-Based Teeth Reconstruction 46

GAUSSIAN DISTRIBUTION

  • All dimensions modeled via multivariate Gaussians

𝒀 = (𝐓 , 𝑼τ, {𝛃𝛖

𝐣 })

𝒒 𝒀

Teeth Prior

slide-47
SLIDE 47

Michael Zollhöfer Model-Based Teeth Reconstruction 47

OVERVIEW

Teeth Prior Model 3D Reconstruction

Scans

Teeth Reconstruction

Contour Extraction Dental Scans Static Images/Video Clip

slide-48
SLIDE 48

Michael Zollhöfer Model-Based Teeth Reconstruction 48

OVERVIEW

Teeth Prior Model 3D Reconstruction

Scans

Teeth Reconstruction

Contour Extraction Dental Scans Static Images/Video Clip

slide-49
SLIDE 49

Michael Zollhöfer Model-Based Teeth Reconstruction 49

CONTOUR EXTRACTION

Scans

slide-50
SLIDE 50

Michael Zollhöfer Model-Based Teeth Reconstruction 50

CONTOUR EXTRACTION

Scans

  • Teeth are featureless
slide-51
SLIDE 51

Michael Zollhöfer Model-Based Teeth Reconstruction 51

CONTOUR EXTRACTION

Scans

  • Teeth are featureless
  • Except the silhouette
slide-52
SLIDE 52

Michael Zollhöfer Model-Based Teeth Reconstruction 52

CONTOUR EXTRACTION

Scans

  • Teeth are featureless
  • Except the silhouette

.

slide-53
SLIDE 53

Michael Zollhöfer Model-Based Teeth Reconstruction 53

CONTOUR EXTRACTION

Scans

  • Boosted Edge Learning (BEL)
  • Teeth are featureless
  • Except the silhouette

.

slide-54
SLIDE 54

Michael Zollhöfer Model-Based Teeth Reconstruction 54

FEATURE FUSION

Scans

slide-55
SLIDE 55

Michael Zollhöfer Model-Based Teeth Reconstruction 55

FEATURE FUSION

Scans

Input

slide-56
SLIDE 56

Michael Zollhöfer Model-Based Teeth Reconstruction 56

FEATURE FUSION

Scans

Input BEL

slide-57
SLIDE 57

Michael Zollhöfer Model-Based Teeth Reconstruction 57

FEATURE FUSION

Scans

Input BEL Gabor

slide-58
SLIDE 58

Michael Zollhöfer Model-Based Teeth Reconstruction 58

FEATURE FUSION

Scans

Input BEL Gabor Fusion

slide-59
SLIDE 59

Michael Zollhöfer Model-Based Teeth Reconstruction 59

FEATURE FUSION

Scans

Input BEL Gabor Fusion Formulas in the paper!

slide-60
SLIDE 60

Michael Zollhöfer Model-Based Teeth Reconstruction 60

OVERVIEW

Teeth Prior Model 3D Reconstruction

Scans

Teeth Reconstruction

Contour Extraction Dental Scans Static Images/Video Clip

slide-61
SLIDE 61

Michael Zollhöfer Model-Based Teeth Reconstruction 61

OVERVIEW

Teeth Prior Model 3D Reconstruction

Scans

Teeth Reconstruction

Contour Extraction Dental Scans Static Images/Video Clip

slide-62
SLIDE 62

Michael Zollhöfer Model-Based Teeth Reconstruction 62

  • Given detected contours 𝑫

TEETH FITTING

slide-63
SLIDE 63

Michael Zollhöfer Model-Based Teeth Reconstruction 63

  • Given detected contours 𝑫
  • Find parameters: 𝒀 = (𝐓 , 𝑼τ, {𝛃𝛖

𝐣 })

TEETH FITTING

slide-64
SLIDE 64

Michael Zollhöfer Model-Based Teeth Reconstruction 64

  • Given detected contours 𝑫
  • Find parameters: 𝒀 = (𝐓 , 𝑼τ, {𝛃𝛖

𝐣 })

𝒀∗ = 𝒃𝒔𝒉 𝐧𝐛𝐲

𝒀

𝒒 𝒀 𝑫 = 𝒃𝒔𝒉 𝐧𝐛𝐲

𝒀 [ 𝒒 𝑫 𝒀

⋅ 𝒒 𝒀 ]

MAXIMUM A POSTERIORI (MAP) PROBLEM

slide-65
SLIDE 65

Michael Zollhöfer Model-Based Teeth Reconstruction 65

  • Given detected contours 𝑫
  • Find parameters: 𝒀 = (𝐓 , 𝑼τ, {𝛃𝛖

𝐣 })

𝒀∗ = 𝒃𝒔𝒉 𝐧𝐛𝐲

𝒀

𝒒 𝒀 𝑫 = 𝒃𝒔𝒉 𝐧𝐛𝐲

𝒀 [ 𝒒 𝑫 𝒀

⋅ 𝒒 𝒀 ]

Teeth Edge Likelihood

MAXIMUM A POSTERIORI (MAP) PROBLEM

slide-66
SLIDE 66

Michael Zollhöfer Model-Based Teeth Reconstruction 66

  • Given detected contours 𝑫
  • Find parameters: 𝒀 = (𝐓 , 𝑼τ, {𝛃𝛖

𝐣 })

𝒀∗ = 𝒃𝒔𝒉 𝐧𝐛𝐲

𝒀

𝒒 𝒀 𝑫 = 𝒃𝒔𝒉 𝐧𝐛𝐲

𝒀 [ 𝒒 𝑫 𝒀

⋅ 𝒒 𝒀 ]

Teeth Edge Likelihood Teeth Prior

MAXIMUM A POSTERIORI (MAP) PROBLEM

slide-67
SLIDE 67

Michael Zollhöfer Model-Based Teeth Reconstruction 67

  • Given detected contours 𝑫
  • Find parameters: 𝒀 = (𝐓 , 𝑼τ, {𝛃𝛖

𝐣 })

𝒀∗ = 𝒃𝒔𝒉 𝐧𝐛𝐲

𝒀

𝒒 𝒀 𝑫 = 𝒃𝒔𝒉 𝐧𝐛𝐲

𝒀 [ 𝒒 𝑫 𝒀

⋅ 𝒒 𝒀 ]

Teeth Edge Likelihood Teeth Prior

MAXIMUM A POSTERIORI (MAP) PROBLEM

Solve with EM-Algorithm

slide-68
SLIDE 68

Michael Zollhöfer Model-Based Teeth Reconstruction 68

EM ALGORITHM

slide-69
SLIDE 69

Michael Zollhöfer Model-Based Teeth Reconstruction 69

EM ALGORITHM

slide-70
SLIDE 70

Michael Zollhöfer Model-Based Teeth Reconstruction 70

EM ALGORITHM

slide-71
SLIDE 71

Michael Zollhöfer Model-Based Teeth Reconstruction 71

EM ALGORITHM

slide-72
SLIDE 72

Michael Zollhöfer Model-Based Teeth Reconstruction 72

EM ALGORITHM

slide-73
SLIDE 73

Michael Zollhöfer Model-Based Teeth Reconstruction 73

EM ALGORITHM

slide-74
SLIDE 74

Michael Zollhöfer Model-Based Teeth Reconstruction 74

EM ALGORITHM

slide-75
SLIDE 75

Michael Zollhöfer Model-Based Teeth Reconstruction 75

E-STEP

slide-76
SLIDE 76

Michael Zollhöfer Model-Based Teeth Reconstruction 76

TEETH EDGE LIKELIHOOD

𝒒 𝑫 𝒀

Teeth Edge Likelihood

slide-77
SLIDE 77

Michael Zollhöfer Model-Based Teeth Reconstruction 77

TEETH EDGE LIKELIHOOD

𝒒 𝑫 𝒀

Teeth Edge Likelihood

𝒒𝒒 𝑫 𝒀 =

slide-78
SLIDE 78

Michael Zollhöfer Model-Based Teeth Reconstruction 78

TEETH EDGE LIKELIHOOD

𝒒 𝑫 𝒀

Teeth Edge Likelihood

𝒒𝒒 𝑫 𝒀 𝒒𝒐 𝑫 𝒀 =

⋅

slide-79
SLIDE 79

Michael Zollhöfer Model-Based Teeth Reconstruction 79

TEETH EDGE LIKELIHOOD

𝒒 𝑫 𝒀

Teeth Edge Likelihood

𝒒𝒒 𝑫 𝒀 𝒒𝒐 𝑫 𝒀 =

⋅

Formulas in the paper!

slide-80
SLIDE 80

Michael Zollhöfer Model-Based Teeth Reconstruction 80

M-STEP

slide-81
SLIDE 81

Michael Zollhöfer Model-Based Teeth Reconstruction 81

M-STEP

slide-82
SLIDE 82

Michael Zollhöfer Model-Based Teeth Reconstruction 82

ITERATE

slide-83
SLIDE 83

Michael Zollhöfer Model-Based Teeth Reconstruction 83

DONE

slide-84
SLIDE 84

Michael Zollhöfer Model-Based Teeth Reconstruction 84

COLOR AND GUMS

slide-85
SLIDE 85

Michael Zollhöfer Model-Based Teeth Reconstruction 85

COLOR AND GUMS

  • Projective Texturing
slide-86
SLIDE 86

Michael Zollhöfer Model-Based Teeth Reconstruction 86

Template Gum Line

COLOR AND GUMS

  • Projective Texturing
  • Fit 3D gum template via Laplacian deformation
slide-87
SLIDE 87

Michael Zollhöfer Model-Based Teeth Reconstruction 87

Template Gum Line Result

COLOR AND GUMS

  • Projective Texturing
  • Fit 3D gum template via Laplacian deformation
slide-88
SLIDE 88

Michael Zollhöfer Model-Based Teeth Reconstruction 88

OVERVIEW

Teeth Prior Model 3D Reconstruction

Scans

Teeth Reconstruction

Contour Extraction Dental Scans Static Images/Video Clip

slide-89
SLIDE 89

Michael Zollhöfer Model-Based Teeth Reconstruction 89

OVERVIEW

Teeth Prior Model 3D Reconstruction

Static Images/Video Clip

Teeth Reconstruction

Contour Extraction Dental Scans

slide-90
SLIDE 90

Michael Zollhöfer Model-Based Teeth Reconstruction 90

MULTI-VIEW IMAGES

slide-91
SLIDE 91

Michael Zollhöfer Model-Based Teeth Reconstruction 91

MONOCULAR VIDEO

slide-92
SLIDE 92

Michael Zollhöfer Model-Based Teeth Reconstruction 92

APPLICATION: TOOTH RESTORATION

1 tooth missing 2 teeth missing 4 teeth missing

slide-93
SLIDE 93

Michael Zollhöfer Model-Based Teeth Reconstruction 93

APPLICATION: TOOTH RESTORATION

1 tooth missing 2 teeth missing 4 teeth missing

slide-94
SLIDE 94

Michael Zollhöfer Model-Based Teeth Reconstruction 94

APPLICATION: TOOTH RESTORATION

1 tooth missing 2 teeth missing 4 teeth missing

slide-95
SLIDE 95

Michael Zollhöfer Model-Based Teeth Reconstruction 95

APPLICATION: TOOTH RESTORATION

1 tooth missing 2 teeth missing 4 teeth missing

slide-96
SLIDE 96

Michael Zollhöfer Model-Based Teeth Reconstruction 96

First non-invasive teeth reconstruction approach

CONCLUSION

slide-97
SLIDE 97

Michael Zollhöfer Model-Based Teeth Reconstruction 97

First non-invasive teeth reconstruction approach Content creation (i.e. digital actor)

CONCLUSION

slide-98
SLIDE 98

Michael Zollhöfer Model-Based Teeth Reconstruction 98

First non-invasive teeth reconstruction approach Content creation (i.e. digital actor) Dentistry (i.e tooth restoration)

CONCLUSION

slide-99
SLIDE 99

THANK YOU!

  • P. Garrido
  • M. Zollhöfer
  • C. Wu
  • D. Bradley
  • T. Beeler
  • C. Theobalt
  • M. Gross
slide-100
SLIDE 100

Michael Zollhöfer Model-Based Teeth Reconstruction 100

APPENDIX