SLIDE 1
Eve EventC tCap: High-Speed Human Motion Capture using an Event Camera
Lan XU 许岚
Hong Kong University of Science and Technology 2020/06/11
Background
2
tCap : High-Speed Human Motion Capture using Eve EventC an Event - - PowerPoint PPT Presentation
tCap : High-Speed Human Motion Capture using Eve EventC an Event - - PowerPoint PPT Presentation
tCap : High-Speed Human Motion Capture using Eve EventC an Event Camera Lan XU Hong Kong University of Science and Technology 2020/06/11 Background 2 1. Background q Previous MoCap systems 2 nd Generation 3 rd Generation 1 st
SLIDE 2
SLIDE 3
- 1. Background
q Previous MoCap systems 1st Generation 2nd Generation 3rd Generation Marker-based MoCap :
- Only reconstruct makers
- Intrusive, restricted clothing
- Not ready for daily usage
High-end marker-less system:
- Many, many cameras
- Green background & fixed space
- Tedious synchronization, calibration
Convenient Capture
- Handheld or single-view
- Consumer-level
- Still fixed captured volume
Technological Trend: Realtime, convenient and high quality 4D human reconstruction is critical
3
SLIDE 4
- 1. Background
q Bottleneck of high-speed human MoCap
- high speed motion analysis is rare
- RGB/RGBD: good lighting for high
frame rates
- Throughput: a VGA RGB stream at
1000 fps for 60 s à 51.5 GB !!!
UnstructuredFusion RobustFusion MonoPerfCap LiveCap
4
SLIDE 5
- 1. Background
q Bottleneck of high-speed human MoCap
- high speed motion analysis is rare
- RGB/RGBD: good lighting for high
frame rates
- Throughput: a VGA RGB stream at
1000 fps for 60 s à 51.5 GB !!! Could we liberates these constraints ?
EventCap: use new device !!!
5
SLIDE 6
Key idea
6
SLIDE 7
7
q Basic idea
- Capturing high-speed human motions at 1000 fps
t_1 t_2 ……
t0_0 t0_2 t0_n
t_0
q Benefits:
- High temporal resolution, HDR (140 dB), low data bandwidth
q Challenges:
- Images & events: unstructured temporal information
- Severe image blur
- 2. Human Modelling: EventCap
SLIDE 8
8
q High-speed human motions
- 2. Human Modelling: EventCap
SLIDE 9
V.S.
High speed camera: Sony RX0 Event camera: DAVIS240C
Only 3.4% data bandwidth
- 2. Human Modelling: EventCap
9
SLIDE 10
10
- 2. Human Modelling: EventCap
q Reconstruction results for sports analysis
Low FPS image Event polarity Reference view in Sony Camera
SLIDE 11
11
q Results of capturing a Ninja in the dark
- Thanks to the high dynamic range (140 dB) of the event camera
Low FPS image Event polarity Reference view in Sony Camera
(Gamma enhancement) (Original images)
- 2. Human Modelling: EventCap
SLIDE 12
Method
12
SLIDE 13
Rigged Template Event stream Intensity image stream DAVIS 240C
- 3. Algorithm details
q Input of EventCap
13
SLIDE 14
- 1. Event Trajectory Generation
- 2. Batch Optimization
- 3. Event-based Pose Refinement
t
2D features at tracking fps Event trajectory Detection !"#$%&
…
Boundary information Events
… …
- 3. Algorithm details
q Framework
14
SLIDE 15
t
2D features at tracking fps Event trajectory Detection !"#$%&
…
Boundary information Events
… …
- 3. Algorithm details
q Stage I: Event Trajectory Generation
15
SLIDE 16
t
2D features at tracking fps Event trajectory Detection !"#$%&
…
Boundary information Events
… …
+ =
Event trajectories Intensity image stream Event stream
q Stage I: Event Trajectory Generation
- 3. Algorithm details
16
SLIDE 17
t
2D features at tracking fps Event trajectory Detection !"#$%&
…
Boundary information Events
… …
- 3. Algorithm details
q Stage I: Event Trajectory Generation
- 2D feature trajectory between adjacent images
- Forward & backward alignment
- Trajectory slicing à 2D correspondence pairs
17
SLIDE 18
t
2D features at tracking fps Event trajectory Detection !"#$%&
…
Boundary information Events
… …
- 3. Algorithm details
q Stage II: Batch Optimization
18
SLIDE 19
t
2D features at tracking fps Event trajectory Detection !"#$%&
…
Boundary information Events
… …
+ = +
2D detection 3D detection Event trajectories
- 3. Algorithm details
q Stage II: Batch Optimization
19
SLIDE 20
t
2D features at tracking fps Event trajectory Detection !"#$%&
…
Boundary information Events
… …
- 3. Algorithm details
q Stage II: Batch Optimization
20
SLIDE 21
t
2D features at tracking fps Event trajectory Detection !"#$%&
…
Boundary information Events
… …
- 3. Algorithm details
q Stage II: Event-based Pose Refinement
21
SLIDE 22
t
2D features at tracking fps Event trajectory Detection !"#$%&
…
Boundary information Events
… …
Results of Stage II Our final results
1 Normalized distance map Event stream
q Stage II: Event-based Pose Refinement
- 3. Algorithm details
22
SLIDE 23
t
2D features at tracking fps Event trajectory Detection !"#$%&
…
Boundary information Events
… …
Results of Stage II Our final results
1 Normalized distance map Event stream
q Stage II: Event-based Pose Refinement
- 3. Algorithm details
23
SLIDE 24
t
2D features at tracking fps Event trajectory Detection !"#$%&
…
Boundary information Events
… …
- 3. Algorithm details
q Stage II: Event-based Pose Refinement
Distance map Before refinement After refinement 24
SLIDE 25
Results
25
SLIDE 26
26
SLIDE 27
q Reconstruction results
27
- 4. Results of EventCap
https://www.xu-lan.com/research.html
SLIDE 28
Summary
28
SLIDE 29
- 5. Future Vision of Human Modeling
q Aspect of MoCap Date:
Figure from Prof. Yaser Sheikh, CMU
Year Throughput (GBps)
The Future
29
EventCap
SLIDE 30
Thanks for your attention!
30
SLIDE 31
The End
31