Real Virtual Humans Gerard Pons-Moll Max Planck Institute for - - PowerPoint PPT Presentation

▶
real virtual humans
SMART_READER_LITE
LIVE PREVIEW

Real Virtual Humans Gerard Pons-Moll Max Planck Institute for - - PowerPoint PPT Presentation

Aug 19, 2022 317 likes •1.76k views

Real Virtual Humans Gerard Pons-Moll Max Planck Institute for Informatics

slide-1
SLIDE 1

Real Virtual Humans

Gerard Pons-Moll

Max Planck Institute for Informatics

slide-2
SLIDE 2
slide-3
SLIDE 3
slide-4
SLIDE 4
slide-5
SLIDE 5
slide-6
SLIDE 6
slide-7
SLIDE 7
slide-8
SLIDE 8
  • M(

θ, β, u; Φ)

slide-9
SLIDE 9
  • M(0, β)

M(θ, 0) M(θ, β) R · M(θ, β) M(θ, β)

slide-10
SLIDE 10
slide-11
SLIDE 11
  • R0, t0

Rj, tj

Xpose = {R0, t0, . . . RN, tN}

slide-12
SLIDE 12
slide-13
SLIDE 13
slide-14
SLIDE 14
  • ωj
  • ωj
  • ωj
  • ωj
  • ω

j

R = e

  • ω = I +

¯ ωj sin( ωj) + ¯ ω

2(1 − cos(

ωj)

slide-15
SLIDE 15
  • ωj
  • ωj
  • j

G( ω, j) =

  • [e

ω]3×3

j3×1 01×3 1

slide-16
SLIDE 16
  • ω1
  • ω2

j1 j2

pb

slide-17
SLIDE 17
  • ω2
  • ω1
  • ω2

j1 j2

pb

slide-18
SLIDE 18
  • ω1
  • ω2
  • ω1
  • ω2

j1 j2

pb

¯ ps = G( ω1, ω2, j1, j2) = G( ω1, j1)G( ω2, j2)¯ pb

2 2

slide-19
SLIDE 19
  • θ = (

ω1, . . . , ωk)T

J = (j1, . . . , jK)T

  • j1

jK

T

slide-20
SLIDE 20
  • θ
slide-21
SLIDE 21
slide-22
SLIDE 22
slide-23
SLIDE 23
  • T ∈ R3N

J ∈ R3K W ∈ RN×K

  • θ ∈ R3K

3N

slide-24
SLIDE 24
  • T ∈ R3N

J ∈ R3K W ∈ RN×K

  • θ ∈ R3K

3K

slide-25
SLIDE 25
  • T ∈ R3N

J ∈ R3K W ∈ RN×K

  • θ ∈ R3K

N×K

slide-26
SLIDE 26
  • T ∈ R3N

J ∈ R3K W ∈ RN×K

  • θ ∈ R3K

3K

slide-27
SLIDE 27
  • T ∈ R3N

J ∈ R3K W ∈ RN×K

  • θ ∈ R3K

W(T, J, W, θ) → vertices

slide-28
SLIDE 28
slide-29
SLIDE 29
slide-30
SLIDE 30
  • P = vec(

⎡ ⎢ ⎢ ⎢ ⎢ ⎣ ∆x1 ∆y1 ∆z1 . . . . . . ∆xN ∆yN ∆zN ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ ) ∈ R3N

slide-31
SLIDE 31
slide-32
SLIDE 32
slide-33
SLIDE 33
  • T ∈ R3N

J ∈ R3K W ∈ RN×K

BP ( θ′)

slide-34
SLIDE 34
slide-35
SLIDE 35
slide-36
SLIDE 36
slide-37
SLIDE 37
slide-38
SLIDE 38
  • Latent parameters → vertices

M(θ, β; w) : R|θ|+|β| → R3N

slide-39
SLIDE 39
slide-40
SLIDE 40
slide-41
SLIDE 41
slide-42
SLIDE 42
slide-43
SLIDE 43
slide-44
SLIDE 44
  • W(T, J, W,

θ) → vertices

M( θ, β) = W(TF ( β, θ), J( β), W, θ) → vertices

  • β
  • θ

F

slide-45
SLIDE 45
  • W(T, J, W,

θ)

W(T(θ), J, W, θ)

slide-46
SLIDE 46
slide-47
SLIDE 47
  • W(T(θ), J, W,

θ) → vertices T( θ) = T + BP ( θ) BP ( θ) =

|f( θ)|

  • i

fi( θ)Pi

slide-48
SLIDE 48
  • ?

BP ( θ) =

|f( θ)|

  • i

fi( θ)Pi

f( θ)

f( θ) = θ

slide-49
SLIDE 49
slide-50
SLIDE 50
  • f(

θ) f( θ)

  • BP (

θ) =

|f( θ)|

  • i

fi( θ)Pi

slide-51
SLIDE 51
  • θ = (

ω1, . . . , ωk)T

  • f(

θ) = [¯ eˆ

ω1 1,1 . . . ¯

eˆ

ω1 3,3

. . . ¯ eˆ

ωK 1,1 . . . ¯

eˆ

ωK 3,3 ]

eˆ

ω1 − I

eˆ

ωK − I

BP ( θ) =

|f( θ)|

  • i

fi( θ)Pi

slide-52
SLIDE 52
slide-53
SLIDE 53
  • J

J = J(T; J ) = J T

slide-54
SLIDE 54
slide-55
SLIDE 55
slide-56
SLIDE 56
  • W(T, J, W,

θ) → vertices

M( θ, β) = W(TF ( β, θ), J( β), W, θ) → vertices

  • β
  • θ

F

slide-57
SLIDE 57
  • M(

θ, β; T, S, P, W, J )

pose shape Input Model parameters to be learned from data

S P W T J

Template (average shape) Shape blend shape matrix Pose blend shape matrix Blendweights matrix Joint regressor matrix

slide-58
SLIDE 58
slide-59
SLIDE 59
  • w = arg min

w

  • j

M( θ, β; w) − 2

slide-60
SLIDE 60
  • …
  • V1

V2 . . . VNsubj

  • = T +
  • S1

S2 . . . SNsubj

  • B

1 2

slide-61
SLIDE 61
  • …
  • V1

V2 . . . VNsubj

  • ≈ T + SB
slide-62
SLIDE 62
slide-63
SLIDE 63
slide-64
SLIDE 64
slide-65
SLIDE 65
slide-66
SLIDE 66
slide-67
SLIDE 67

−

  • E(v) =
  • si∈S

dist(si, A(v)) + Eprior(v)

slide-68
SLIDE 68

−

  • E(θ, β) =
  • si∈S

dist(si, M(θ, β)) + Eprior(θ, β)

slide-69
SLIDE 69

E(θ, β, v) =

  • si∈S

dist(si, A(v)) + dist(A(v), M(θ, β)) + Eprior(θ, β)

− − +

slide-70
SLIDE 70

http://dfaust.is.tue.mpg.de

slide-71
SLIDE 71
slide-72
SLIDE 72
slide-73
SLIDE 73
  • θ

β

  • c

M(θ, β, c)

slide-74
SLIDE 74
slide-75
SLIDE 75
slide-76
SLIDE 76
slide-77
SLIDE 77
slide-78
SLIDE 78
slide-79
SLIDE 79
  • Single frame

registration Segmentation Multi-part registration Input: scans + garment priors

slide-80
SLIDE 80
slide-81
SLIDE 81
slide-82
SLIDE 82
  • Cloth template

SMPL Scan

slide-83
SLIDE 83
  • Cloth

Template

slide-84
SLIDE 84
  • Cloth

Template

slide-85
SLIDE 85

E(θ, β, v) = Edata(v) + Ecpl(θ, β, v) +

slide-86
SLIDE 86

E(θ, β, v) = Edata(v) + Ecpl(θ, β, v) + + Eboundary(v) + Elap(v)

slide-87
SLIDE 87
slide-88
SLIDE 88
slide-89
SLIDE 89
slide-90
SLIDE 90
slide-91
SLIDE 91
slide-92
SLIDE 92
slide-93
SLIDE 93
  • Zhang et al. CVPR’17. BUFF dataset: http://buff.is.tue.mpg.de
slide-94
SLIDE 94
slide-95
SLIDE 95
slide-96
SLIDE 96
slide-97
SLIDE 97
  • arg min

θ,β dist(ˆ

z (M(θ, β)), z)

  • P(·)

z

slide-98
SLIDE 98
  • arg min

θ,β dist(ˆ

z (M(θ, β)), z)

P(·)

  • z
slide-99
SLIDE 99
  • arg min

θ,β dist(ˆ

z (M(θ, β)), z)

slide-100
SLIDE 100
  • θ

β

  • w
slide-101
SLIDE 101
slide-102
SLIDE 102
slide-103
SLIDE 103
slide-104
SLIDE 104
slide-105
SLIDE 105
slide-106
SLIDE 106
slide-107
SLIDE 107
slide-108
SLIDE 108
slide-109
SLIDE 109
slide-110
SLIDE 110
slide-111
SLIDE 111
slide-112
SLIDE 112
  • arg min

β,d Econs(β, d)

slide-113
SLIDE 113
slide-114
SLIDE 114
slide-115
SLIDE 115
slide-116
SLIDE 116
slide-117
SLIDE 117
slide-118
SLIDE 118
slide-119
SLIDE 119
slide-120
SLIDE 120
slide-121
SLIDE 121
  • M(θ,β)
  • θ

β

  • P(·)
  • w

arg min

w ˆ

z(I, w) − z z

slide-122
SLIDE 122
slide-123
SLIDE 123
slide-124
SLIDE 124
slide-125
SLIDE 125
slide-126
SLIDE 126
slide-127
SLIDE 127
slide-128
SLIDE 128
slide-129
SLIDE 129
slide-130
SLIDE 130
slide-131
SLIDE 131
slide-132
SLIDE 132
slide-133
SLIDE 133
  • HumanEva

Human3.6M MPII human pose

slide-134
SLIDE 134
slide-135
SLIDE 135
slide-136
SLIDE 136
slide-137
SLIDE 137
slide-138
SLIDE 138
slide-139
SLIDE 139
slide-140
SLIDE 140
slide-141
SLIDE 141
slide-142
SLIDE 142
slide-143
SLIDE 143