3D Scanning and Reconstruction (02) RNDr. Martin Madaras, PhD. - - PowerPoint PPT Presentation

▶
3d scanning and reconstruction 02
SMART_READER_LITE
LIVE PREVIEW

3D Scanning and Reconstruction (02) RNDr. Martin Madaras, PhD. - - PowerPoint PPT Presentation

May 11, 2023 173 likes •1.24k views

Virtual and Augmented Reality 3D Scanning and Reconstruction (02) RNDr. Martin Madaras, PhD. madaras@skeletex.xyz How the lectures should look like #1 Ask questions, please!!! - Be communicative - www.slido.com #VAR01 - More active you

slide-1
SLIDE 1

Virtual and Augmented Reality

3D Scanning and Reconstruction (02)

  • RNDr. Martin Madaras, PhD.

madaras@skeletex.xyz

slide-2
SLIDE 2

2

  • Ask questions, please!!!
  • Be communicative
  • www.slido.com #VAR01
  • More active you are, the better for you!

How the lectures should look like #1

slide-3
SLIDE 3

3

3D Model Representations

slide-4
SLIDE 4

3D Object Representations

4

 How to represent a 3D object?

slide-5
SLIDE 5

3D Object Representations

5

 Polygons…

slide-6
SLIDE 6

3D Object Representations

6

 Voxels…

slide-7
SLIDE 7

3D Object Representations

7

 Recreating artwork

slide-8
SLIDE 8

3D Object Representations

8

 Mechanical objects

slide-9
SLIDE 9

3D Object Representations

9

 Maps, Cities, Landscape

slide-10
SLIDE 10

3D Object Representations

10

 Clouds, Smoke, Fog, Water

slide-11
SLIDE 11

3D Object Representations

11

 Points

 Range Image, Point Cloud

 Surfaces

 Polygonal, Subdivision, Parametric, Implicit

 Solids

 Voxels, BSP Tree, CSG, Sweep, etc.

 Hierarchical Structures

 Scene graph, Application specific…

slide-12
SLIDE 12

Why so many representation?

12

 Efficiency for different tasks

 Rendering  Acquisition  Manipulation  Animation  Analysis

 Data structures determine algorithms

slide-13
SLIDE 13

Range Image

13

 Set of 3D points mapping to pixels of depth image  Structured Point Cloud

 Acquired using a range scanner (eg. Kinect)

slide-14
SLIDE 14

Point Cloud

14

 Unstructured set of 3D point samples

 Acquired from multiple range scans, vision, etc.

slide-15
SLIDE 15

Polygonal Mesh

15

 Connected mesh of polygons (usually triangles)

 Most common representation, supported in OpenGL

slide-16
SLIDE 16

Key Questions

16

 How to refine mesh ?

 Aim for properties like smoothness

 How to store mesh ?

 Aim for efficiency of implementing subdivision rules

slide-17
SLIDE 17

Polygonal meshes

17

 V, E, F  P, S

slide-18
SLIDE 18

Polygonal Meshes

18

 Mesh Representations

 Independent faces  Vertex and face tables  Adjacency lists  Winged-Edge

slide-19
SLIDE 19

Independent Faces

19

 Each face lists vertex coordinates

 Redundant vertices  No topology information

slide-20
SLIDE 20

Vertex and Face Tables

20

 Each face lists vertex references

 Shared vertices  Still no topology information

slide-21
SLIDE 21

Adjacency Lists

21

 Store all vertex, edge and face adjacency

 Efficient topology traversal  Extra storage

slide-22
SLIDE 22

Partial Adjacency Lists

22

 Can we can store only some adjacency information and

derive others?

slide-23
SLIDE 23

Winged Edge

23

 Adjacency encoded in edges

 All adjacencies in O(1) time  Little extra storage  Arbitrary polygons

slide-24
SLIDE 24

Winged Edge Example

24

slide-25
SLIDE 25

Taxonomy of 3D representations

25

slide-26
SLIDE 26

Computational Differences

26

 Efficiency

 Computational complexity ( O(n log n) )  Space/Time trade-off  Numerical stability/accuracy

 Simplicity

 Hardware acceleration  Ease of acquisition  Software creation and maintenance

 Usability

 Designer vs. computational engine

slide-27
SLIDE 27

Parametric vs. polygonal

27

 Parametric

 smooth, reparametrizable  harder rendering  precise rendering

 Polygonal

 discrete, hard to reparametrize  faster rendering or rasterization  approximative rendering

slide-28
SLIDE 28

28

  • Ask questions, please!!!
  • Be communicative
  • www.slido.com #VAR01
  • More active you are, the better for you!

How the lectures should look like #2

slide-29
SLIDE 29

29

3D Cameras

slide-30
SLIDE 30

“Stereo Vision”

30

 > 2 cameras  Photogrametry

slide-31
SLIDE 31

“Time of Flight”

31

 Ligh pulses and its reflections captured by photodiodes

slide-32
SLIDE 32

“Structured Light”

32

 Projection of patterns, capturing by camera

slide-33
SLIDE 33

“Structured Light”

33

 Binary sequence – position

slide-34
SLIDE 34

34

Scan Registration

 Iterative Closest Point (ICP)

https://www.youtube.com/watch?v=k116t4cef-4

slide-35
SLIDE 35

35

Scan Registration

 Iterative Closest Point (ICP)

slide-36
SLIDE 36

36

Scan Registration

 Iterative Closest Point (ICP)

slide-37
SLIDE 37

37

ICP

 Normal-space Sampling

slide-38
SLIDE 38

38

ICP

 Point-to-Plane Error Metric

 Minimize the sum of the squared distance between a point and

the tangent plane at its correspondence point [Chen & Medioni 91]

slide-39
SLIDE 39

39

Filtering

 Multiview filtering

 Volumetric grid  Sample radius

 ML-based using CNN

slide-40
SLIDE 40

40

Mesh reconstruction

 Conversion into an implicit function

 Poisson reconstruction

slide-41
SLIDE 41

41

Mesh reconstruction

 Isosurface extraction

 Marching cubes

slide-42
SLIDE 42

42

Marching cubes

 Create cubes  Classify vertices  Build indices  Lookup edge list

slide-43
SLIDE 43

43

Marching cubes

 Create cubes  Classify vertices  Build indices  Interpolate Triangle

Vertices

slide-44
SLIDE 44

44

Marching cubes

 Create cubes  Classify vertices  Build indices  Interpolate Triangle

Vertices

 Calculate normals

slide-45
SLIDE 45

45

Marching cubes

slide-46
SLIDE 46

46

Marching cubes

slide-47
SLIDE 47

47

Marching cubes

slide-48
SLIDE 48

48

Marching cubes

slide-49
SLIDE 49

49

Marching cubes

slide-50
SLIDE 50

50

Point cloud Rigid Alignment and Fusion of Scans

PRAFOS

slide-51
SLIDE 51

51

  • Point cloud Rigid Alignment and Fusion of Scans

Rotable

slide-52
SLIDE 52

52

Rotable

slide-53
SLIDE 53

53

TAROS

slide-54
SLIDE 54

54

TAROS

slide-55
SLIDE 55

55

TAROS

slide-56
SLIDE 56

Inspection and Editing

56

Inspection and Editing of Point Clouds in VR Bachelor thesis 2019

  • Bc. Lukáš Gajdošech
slide-57
SLIDE 57

Inspection and Editing

57

slide-58
SLIDE 58

58

Non-Rigid scanning and reconstruction

Human Model Fusion

slide-59
SLIDE 59

Ultimate Goal

59

 Real-time capturing of human performance and

reconstruction rendering in virtual reality

slide-60
SLIDE 60

VR Pipeline

60

 Skeleton and surface tracking  Surface reconstruction fusion  Compression & data streaming  Surface reconstruction from textures and skeleton  Rendering & applications

slide-61
SLIDE 61

61

Skeleton Tracking

slide-62
SLIDE 62

Point Cloud & Tracked Skeleton

62

slide-63
SLIDE 63

Tracking Optical Flow

63

slide-64
SLIDE 64

64

Skeletex Data Structure

slide-65
SLIDE 65

Skeletex

65

Skeletex: Skeleton-texture Co-representation for Topology-driven Real-time Interchange and Manipulation of Surface Regions PACIFIC GRAPHICS 2018 (Martin Madaras, Adam Riečický, Michal Mesároš, Martin Stuchlík, Michal Piovarči)

slide-66
SLIDE 66

Skeletex Pipeline

66

 Automatic conversion of standard mesh

slide-67
SLIDE 67

Skeletex Pipeline

67

 Real time GPU reconstruction  Pose-independent

slide-68
SLIDE 68

Mesh Generation

68

 Regular quad lattice is generated for each segment

slide-69
SLIDE 69

69

Skeletex Construction

slide-70
SLIDE 70

Skeletons

70

 Automatically extracted  Generated by artist  Property

 neighboring surface segments are

neighbors in the skeleton

slide-71
SLIDE 71

Skeletex Layout

71

slide-72
SLIDE 72

Separation Planes

72

 Space segmentation

slide-73
SLIDE 73

Mesh Segmentation

73

 Flood-fill algorithm over vertices

slide-74
SLIDE 74

Skeletex Segmentation

74

slide-75
SLIDE 75

Hybrid Skeletex Parameterization

75

slide-76
SLIDE 76

SPD

76

 Separation Planes Driven

 Deformed piecewise cylindrical parameterization  Given directly by the structure

slide-77
SLIDE 77

VSM

77

 Vertex Sampling Maximizing

 Balances the triangle area in texture space

slide-78
SLIDE 78

Bone Tangent Space

78

 Pose-independent, normalized form of the surface

slide-79
SLIDE 79

Animation support

79

 Roll quaternion

slide-80
SLIDE 80

Animation support

80

 Yaw-pitch Quaternion

slide-81
SLIDE 81

Quaternion Deformations

81

 Roll quaternion  Yaw-pitch quaternion

slide-82
SLIDE 82

Quaternion Deformations

82

slide-83
SLIDE 83

Skeletex LoD

83

 32x32 up to 1024x1024

slide-84
SLIDE 84

Skeletex Reconstructions

84

slide-85
SLIDE 85

Segment Interchange

85

slide-86
SLIDE 86

Texture-space Morphing

86

slide-87
SLIDE 87

Texture-space Morphing

87

slide-88
SLIDE 88

Texture-space Morphing

88

slide-89
SLIDE 89

Texture-space Morphing

89

slide-90
SLIDE 90

Texture-space Morphing

90

slide-91
SLIDE 91

91

Skeletex Fusion

slide-92
SLIDE 92

Skeleton-based Segmentation

92

slide-93
SLIDE 93

Segmentation Map

93

slide-94
SLIDE 94

Height Map

94

slide-95
SLIDE 95

Intensity Map

95

slide-96
SLIDE 96

Fusion

96

slide-97
SLIDE 97

97

State of the art dynamic fusion techniques

slide-98
SLIDE 98

VolumeDeform

98

slide-99
SLIDE 99

Fusion4D

99

slide-100
SLIDE 100

BodyFusion

100

slide-101
SLIDE 101

Our Advantage

101

 All the other state of the art methods are volumetric  The fusion structure is also the streaming structure  Video codec can be directly used

slide-102
SLIDE 102

102

The end

slide-103
SLIDE 103

103

Give feedback for lectures Ask questions Future interest in CG jobs? Seminar every even week Thursday 16 – 18 (15 Oct, 29 Oct, etc) https://meet.google.com/fqn-ykup-bya

Feedback, Questions, Beers !

slide-104
SLIDE 104

104

Acknowledgements

 Thanks to all the people, whose work is shown here and whose

slides were used as a material for creation of these slides:

Matej Novotný, GSVM lectures at FMFI UK Peter Drahoš, PPGSO lectures at FIIT STU Output of all the publications and great team work Very best data from 3D cameras

slide-105
SLIDE 105

105

www.skeletex.xyz madaras@skeletex.xyz

Questions ?!