SLIDE 1
Introduction
What does a continuous sign language recognition (SLR) system do?
word vocabulary: apple, sun, today, catch, you ……
2
today is sunny … SLR system sign video sentence
Brief Introduction to Continuous Sign Language Recognition - - PowerPoint PPT Presentation
Brief Introduction to Continuous Sign Language Recognition - - PowerPoint PPT Presentation
Brief Introduction to Continuous Sign Language Recognition 2019.1.19 Introduction What does a continuous sign language recognition (SLR) system do? word vocabulary: apple, sun, today, catch, you today is SLR system
SLIDE 2
SLIDE 3
Introduction
Evaluation on Continuous SLR
Word Error Rate (WER) For example, prediction: I (have) a cat that named Jerry. groundtruth: I have a cat named Tom. Calculate the WER:
3
1 1 1=0.5 6
SLIDE 4
Introduction
Continuous SLR is weakly-supervised 解决 Continuous SLR 问题的主流思路
受语音识别领域启发：对每一帧识别，合并结果
Connectionist Temporal Classification ( CTC ) CNN-RNN-CTC framework
受机器翻译领域启发：从特征序列映射到文本序列
Encoder-Decoder framework
4
SLIDE 5
Introduction
5
CTC: 逐一识别，再合并
Graves A, Fernández S, Gomez F, et al. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. ICML 2006
SLIDE 6
Recurrent Convolutional Neural Networks for Continuous Sign Language Recognition by Staged Optimization [CVPR 2017]
6
Framework : Spatio-temporal CNN - BLSTM - CTC
SLIDE 7
Recurrent Convolutional Neural Networks for Continuous Sign Language Recognition by Staged Optimization [CVPR 2017]
Step1: end-to-end learning
7
Conv1D: 沿时间维度卷积 (K+1)×N d×N
SLIDE 8
Recurrent Convolutional Neural Networks for Continuous Sign Language Recognition by Staged Optimization [CVPR 2017]
Step2: Feature learning with alignment proposal
alignment proposal: output of BLSTM to finetune the spatio-temporal feature extractor
8
SLIDE 9
Recurrent Convolutional Neural Networks for Continuous Sign Language Recognition by Staged Optimization [CVPR 2017]
Step3: Sequence learning from representations
9
SLIDE 10
Recurrent Convolutional Neural Networks for Continuous Sign Language Recognition by Staged Optimization [CVPR 2017]
10
Experimental results
SLIDE 11
Recurrent Convolutional Neural Networks for Continuous Sign Language Recognition by Staged Optimization [CVPR 2017]
11
Comparisons
SLIDE 12
Recurrent Convolutional Neural Networks for Continuous Sign Language Recognition by Staged Optimization [CVPR 2017]
12
Motivated by this paper…
alignment proposal: probability distribution -> argmax-> word a staged optimization -> more staged optimization ……
SLIDE 13
Connectionist Temporal Fusion for Sign Language Translation [MM2019]
13
SLIDE 14
Temporal COV
14
Connectionist Temporal Fusion for Sign Language Translation [MM2019]
SLIDE 15
Optimization Decoding
argmax-> delete blank -> delete continuous repetitions
15
Connectionist Temporal Fusion for Sign Language Translation [MM2019]
SLIDE 16
experimental result
16
Connectionist Temporal Fusion for Sign Language Translation [MM2019]
SLIDE 17
experimental result
17
Connectionist Temporal Fusion for Sign Language Translation [MM2019]
SLIDE 18
Comparisons
18
Connectionist Temporal Fusion for Sign Language Translation [MM2019]
SLIDE 19