SLIDE 1
AI Animation Team
AI Animation Team Clip1 Clip2 Clip4 Clip3 Clip5 Clip6 Clip7 - - PowerPoint PPT Presentation
AI Animation Team Clip1 Clip2 Clip4 Clip3 Clip5 Clip6 Clip7 - - PowerPoint PPT Presentation
AI Animation Team Clip1 Clip2 Clip4 Clip3 Clip5 Clip6 Clip7 Clip8 clip9 Automatically Music Dance Animation Neural Music Animation Network Noise In Output Lack of Paired Data Paired match 3D Skeleton Animation Music
SLIDE 2
SLIDE 3
SLIDE 4
Clip1 Clip2 Clip5 Clip3 Clip6 Clip8 clip9 Clip4 Clip7
SLIDE 5
Automatically
Music Dance Animation
SLIDE 6
SLIDE 7
Music Animation
Noise In Output Lack of Paired Data Neural Network
SLIDE 8
Choreography Video
huge number of them available match
- 3D Skeleton Animation
- Music
Paired
- 2D Skeleton Animation
- Music
Paired
SLIDE 9
Music Animation
Neural Network Match
2D Animation Feature Pre-recorded 3D Skeleton Animation （3D）
Limited
Break into smaller pieces then reconstruct
“Unlimited”
SLIDE 10
- Position (Pose)
- Velocity
- Acceleration
- Motion Beats
How do we match 2D animation with 3D animation？
Velocity Motion Beats
SLIDE 11
Melody Tabs
16 Beats One Cycle
Choreography Labanotation Phase
0.25 Phase 0.5 Phase 0.75 Phase 1.0 Phase
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Phase
Beat Counts
1.25 Phase 1.5 Phase
SLIDE 12
# 7 # 7 # 6 # 6
Segmented Dance Clips
# 1 # 2 # 3 # 4 # 5 # 2 # 4
- Cut dance cycle into smaller sections
base on motion beats
- Reconstruct dance cycle from these
smaller sections based on phase
SLIDE 13
Music Animation
Neural Network Phase-based Match
2D Animation Feature Pre-recorded 3D Skeleton Animation （3D）
SLIDE 14
Audio Track Phase Based Dance Matching Animation Warping Audio-Motion Interpreter Network Dance Animation Recorded Animation Motion Curve Analysis Dance Clip Segments Dance Motion Feature Dance Motion Feature Matched Dance Sequence Dance Video Train
Dance Matching
System Overview
1 2 3
SLIDE 15
3D Dance Sequence
Audio Track Video Track Choreography Video Phase Based Dance Matching Audio Features Dance Motion Features Matched Sequence
Human Pose Detect Clip Segments Clip Segments
… … 1 hour mocap dataset
Fine Grained Matching
Animation Warping Audio-Motion Interpreter Network … … Choreography Video Playlist
Video Clip Motion Feature Audio Track
train
Phase
System Overview
Intelligent Choreography Pipeline Structure
1 2 3
SLIDE 16
Audio-Motion Interpreter Network
Database Structure
YouTube – Choreography Playlists
~ 200 videos 3-6 minutes / video
- Professional Choreography Studios
- K-pop/Urban/Chinese Traditional
- Varying bpm 120-200
Multiple Dancers
Challenges
Moving Camera Sliced Scenes
2D Pose Extraction 16 hours 36 min 500 hours
SLIDE 17
Audio-Motion Interpreter Network
Automatic Pipeline And the Model Choice
Video Filter Slice and Clean Dance Video Music Beat Info Audio Stream Video Steam 2D Pose Extraction Librosa Aubio Mel-spec CQT Beat Speed Info Mapping Network Conv1D/LSTM+Conv2 D Mixture Density Network ConvNets LSTM Mixture Density Networks Phase-based
- Dancer-focused
- Stable Camera
- Continuous
SLIDE 18
Audio-Motion Interpreter Network
Problem Definition Modelling The Mapping From Audio Feature to Motion Features
Audio Track
Audio Features
Dance Motion Features Audio-Motion Interpreter Network Phase Based Dance Matching
Mel-spec
Zero-Padding
Temporal and Frequency Different Length based on bpm Fixed Length
SLIDE 19
Audio-Motion Interpreter Network
Problem Definition Modelling The Mapping From Audio Feature to Motion Features
Audio Track Audio Features Dance Motion Features Audio-Motion Interpreter Network Phase Based Dance Matching
time type delta angle max vel duration
Mel-spec
Dance Motion Features For each Joint One Motion Curve 8 Major Joints Shoulder/Elbow/Keen/Ankle
SLIDE 20
Audio-Motion Interpreter Network
Problem Definition Modelling The Mapping From Audio Feature to Motion Features
SLIDE 21
Audio-Motion Interpreter Network
Problem Definition Modelling The Mapping From Audio Feature to Motion Features
Audio Track Audio Features Dance Motion Features Audio-Motion Interpreter Network Phase Based Dance Matching Mel-spec Training Time 6 hours 40mins 84hours 40mins
SLIDE 22
Audio-Motion Interpreter Network
Problem Definition Modelling The Mapping From Audio Feature to Motion Features
Audio Track Audio Features Dance Motion Features Audio-Motion Interpreter Network Phase Based Dance Matching Mel-spec Inference Time for 16-beat 18 ms 1500 ms
SLIDE 23
Music Waveform Fragments Mel Spectrograms ConvNets Dance Motion Features New Audio Track Dance Motion Features 3D Dance Sequence Audio-Motion Interpreter Fine Grained Dance Matching Waveform Fragments Audio Track Video Track Online Choreography Video Playlist 2D Motion Motion Curve Analysis Module Training
Audio-Motion Interpreter Network
Network Structure and Training procedure
ConvNets
SLIDE 24
Beats Distribution Beat Max Velocity Beats Delta Distance Thigh Calf Upper-arm Fore-arm Local Angle Speed Curves Global Guideline Curves Local Feature Curves Video Frames Detected 2D Motions
Lower Body Rhythm Upper Body Rhythm Beat Breakpoints
3D Dance Animation Clips 3D Dance
Clip Retrieval
Beat Strength
Motion Curve Analysis
Phase-based Dance Matching
2.1 Motion Curve Analysis and Retrieval Module
SLIDE 25
Melody Tabs
16 Beats One Phase
Choreography Labanotation Phase
0.25 Phase 0.5 Phase 0.75 Phase 1.0 Phase
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Phase
Beat Counts
1.25 Phase 1.5 Phase
Phase-based Dance Matching
2.2 Choreography Phase Circle
SLIDE 26
Inside Phase Breakpoints Distribution
Automatic Dataset Preprocess Segmented Dance Clips
# 1 # 2 # 3 # 4 # 5 # 6 # 7 # 8 Motion Curve Analysis # 1 # 2 # 3 # 4 # 5 # 6 # 7 # 8
Phase-based Clips 256 frames
Fine- grained Slicing Mocap Animation Clip
Phase-based Dance Matching
2.3 Phase-based Fine-grained Data Preparation Phase-based Slicing
Breakpoint #1 Breakpoint #2
SLIDE 27
Segmented Dance Clips
2/16 Bests Dance Phase Breakpoint #1 6/16 Bests Dance Phase Breakpoint #2 12/16 Bests
Breakpoint #3
Dance Phase Breakpoint #0 Clip # 4 Phase # 2 Clip # 2 Segment # 1 Clip # 7 Phase # 3 Clip # 8 Phase # 2 Matched Phase Potential Candidates # 1 # 2 # 3 # 4 # 5 # 6 # 7 # 8 # 2 # 4 # 7 # 8
Phase-based Dance Matching
2.4 Phase-based Fine-grained Motion Matching
Dance Motion Features Audio-Motion Interpreter Network Guided Match
Orientation Pose Similarity Velocity Vector Angel Velocity
SLIDE 28
Original Animation Beat Sequence Music Beat Sequence Adjusted Animation Beat Sequence
1.0x Original Playback Speed 1.0x Adjusted Playback Speed
Time Warp Mechanism
Align Animation Beats with Music Beats
Animation Warping
Enhance the Emotional Impact
SLIDE 29
Adjusted Animation Beat Sequence
Time Warp Mechanism
Animation Warping
Enhance the Emotional Impact
Original Animation Beat Sequence Music Beat Sequence
1.0x Original Playback Speed
Matching Beats According to Strength Align Animation Beats with Music Beats
SLIDE 30
Animation Warping
Enhance the Emotional Impact
Time Warping Curve
SLIDE 31
Animation Warping
Enhance the Emotional Impact
Time Warping Curve Enhanced Beats
SLIDE 32
Animation Warping
Enhance the Emotional Impact
Before Beats Aligned
SLIDE 33
Huge Pipeline Efficiency Boost
From GPU Computation
Audio Track Phase Based Dance Matching Animation Warping Audio-Motion Interpreter Network Dance Animation Dance Motion Feature Matched Dance Sequence Dance Video 2D Pose Extraction For Training
1 2
Training Time Phase 0 16 hour Phase 1 6 hour Phase 2 2 hour Phase 0 500 hour Phase 1 84 hour Phase 2 20 hour
SLIDE 34
Music > {neural network}>{matching}> Animation
System Performance on UGC platform
SLIDE 35
Dance Matching
Online Feedback
10000000+ request UGC Widely Spread
in Multiple Video Platform
SLIDE 36
Dance Matching
Online UGC on Bilibiili
SLIDE 37
This System is Currently