Fine-Grained Similarity Measurement of Educational Videos and - PowerPoint PPT Presentation

Fine-Grained Similarity Measurement of Educational Videos and Exercises Xin Wang 1 , Wei Huang 1 , Qi Liu 1 *, Yu Yin 1 , Zhenya Huang 1 , Le Wu 2 , Jianhui Ma 1 , Xue Wang 3 1 University of Science and Technology of China 2 Hefei University of Technology 3 Nankai University Anhui Province Key Lab. of Big Data Analysis and Application, University of S&T of China BDAA, USTC

01 Introduction Anhui Province Key Lab. of Big Data Analysis and Application, University of S&T of China BDAA, USTC

Introduction Ø Related Content Recommendation Related content recommendation on Khan Academy Anhui Province Key Lab. of Big Data Analysis and Application, University of S&T of China BDAA, USTC

Introduction Ø Partial Similar An example from Khan Academy Q1: Are they similar? Q2: Which segments are similar to this exercise? (fine-grained) Anhui Province Key Lab. of Big Data Analysis and Application, University of S&T of China BDAA, USTC

Introduction Ø Fine-Grained Similarity Measurement Input Model Output Anhui Province Key Lab. of Big Data Analysis and Application, University of S&T of China BDAA, USTC

02 Research Contents Anhui Province Key Lab. of Big Data Analysis and Application, University of S&T of China BDAA, USTC

Challenge 1 Ø How to model the multimodal segment? Captions • Keyframes • Ø Spatial and Temporal Information Segment Representation Network Anhui Province Key Lab. of Big Data Analysis and Application, University of S&T of China BDAA, USTC

Challenge 2 Ø Semantic Associations among Video Segments Anhui Province Key Lab. of Big Data Analysis and Application, University of S&T of China BDAA, USTC

Challenge 2 Ø How to model the semantic associations between adjacent video segments? Multiscale Perceptual Fusion Anhui Province Key Lab. of Big Data Analysis and Application, University of S&T of China BDAA, USTC

Challenge 3 Ø How to learn the fine-grained similarity by just exploiting the video-level labeled data? The segment-level labeled data is scarce and costly. • The video-level labeled data is much easier to obtain. • ! " similar exercise ! #" dissimilar exercise $ margin distance % regularization hyperparameter Anhui Province Key Lab. of Big Data Analysis and Application, University of S&T of China BDAA, USTC

03 Experiments Anhui Province Key Lab. of Big Data Analysis and Application, University of S&T of China BDAA, USTC

Dataset All the data were crawled from the Khan Academy’s math domain ( https://www.khanacademy.org/math ) Anhui Province Key Lab. of Big Data Analysis and Application, University of S&T of China BDAA, USTC

Results Anhui Province Key Lab. of Big Data Analysis and Application, University of S&T of China BDAA, USTC

Ablation Experiments Ø Visual Information is helpful Ø Textual Information is more important Ø All the key modules are eddective Anhui Province Key Lab. of Big Data Analysis and Application, University of S&T of China BDAA, USTC

Case Study Anhui Province Key Lab. of Big Data Analysis and Application, University of S&T of China BDAA, USTC

Thanks for Listening! Anhui Province Key Lab. of Big Data Analysis and Application, University of S&T of China BDAA, USTC

ACM MM 2020 QA Session Fine-Grained Similarity Measurement of Educational Videos and Exercises Any Questions? Just be free to let me know! Anhui Province Key Lab. of Big Data Analysis and Application, University of S&T of China BDAA, USTC