VideoForest: Interactive Visual Summarization of Video Streams Based - PowerPoint PPT Presentation

VideoForest: Interactive Visual Summarization of Video Streams Based on Danmu Data Zhida Sun † , Mingfei Sun † , Nan Cao ‡ , Xiaojuan Ma † † Department of Computer Science and Engineering, Hong Kong University of Science and Technology ‡ College of Design and Innovation, TongJi University

Outline • Motivation • System Architecture • System overview • Preprocessing • Analysis • Visualization • • Evaluation • Conclusion 2/24/2017 VideoForest, HCI Initiative, HKUST 2

Video Summarization on Youtube Title Description Screenshot Summary Category Review 2/24/2017 VideoForest, HCI Initiative, HKUST 3

Summarization of Video Streams • Internal summarization techniques Video, audio, linguistic features • • External summarization techniques Audience’s preferences and interests • Video co-summarization [Chu et al. 2015] User preferences based [Kannan et al. 2013] 2/24/2017 VideoForest, HCI Initiative, HKUST 4

Summarization of Video Streams • Summarization representation Matrix form [Lu and Grauman 2013] • Illustration: Visual storyline & Imagehive • Imagehive [Tan et al., 2011] Visual storylines [Chen et al., 2012] 2/24/2017 VideoForest, HCI Initiative, HKUST 5

Live Streaming Platforms 2/24/2017 VideoForest, HCI Initiative, HKUST 6

Video Platforms with Danmu Danmaku screen capture Danmu screen capture 2/24/2017 VideoForest, HCI Initiative, HKUST 7

System Architecture Goal : Video + Danmu + Interaction  Summarization 2/24/2017 VideoForest, HCI Initiative, HKUST 8

Video Demo 2/24/2017 VideoForest, HCI Initiative, HKUST 9

Processing 2/24/2017 VideoForest, HCI Initiative, HKUST 10

Feature Extraction • Visual feature extraction Key frames extraction from compressed video • Hierarchical structure of a video sequence [Mendi and Bayrak 2010] 2/24/2017 VideoForest, HCI Initiative, HKUST 11

Feature extraction • Textual feature extraction syntactic and semantic analysis • Similarity Sentiment Topic Function Symbol Pattern 2/24/2017 VideoForest, HCI Initiative, HKUST 12

Analysis 2/24/2017 VideoForest, HCI Initiative, HKUST 13

Meta-frame, Session and Scene Cluster … Meta-frame • Visual attributes: image content • Textual attributes: audience’s reaction 2/24/2017 VideoForest, HCI Initiative, HKUST 14

Meta-frame, Session and Scene Cluster Session • Color palette of the entire session • Synoptic linguistic attributes of danmu in the time range 2/24/2017 VideoForest, HCI Initiative, HKUST 15

Meta-frame, Session and Scene Cluster Scene cluster • Danmu commentary density 2/24/2017 VideoForest, HCI Initiative, HKUST 16

Visualization 2/24/2017 VideoForest, HCI Initiative, HKUST 17

System Interface Overview 2/24/2017 VideoForest, HCI Initiative, HKUST 18

Interactions Highlighting Smart summarization Scene preview New session generating View switching 2/24/2017 VideoForest, HCI Initiative, HKUST 19

Visual Metaphor and Encoding 2/24/2017 VideoForest, HCI Initiative, HKUST 20

Session Summary View 2/24/2017 VideoForest, HCI Initiative, HKUST 21

Meta-frame aggregation 2/24/2017 VideoForest, HCI Initiative, HKUST 22

Circle Image Packing 2/24/2017 VideoForest, HCI Initiative, HKUST 23

Storyline threading 2/24/2017 VideoForest, HCI Initiative, HKUST 24

Case study • 205 seconds long • 2052 danmu posts • 61 I-frames • 1804 P-frames • 61 sessions • 9 scenes 2/24/2017 VideoForest, HCI Initiative, HKUST 25

User observation • Thirteen experts • Live demo • Interview & open discussion Insights • Applications • 2/24/2017 VideoForest, HCI Initiative, HKUST 26

Future Work and Conclusion • Conclusion • Future work Long video stream scalability • • Hierarchical aggregation Advanced video analysis techniques • 2/24/2017 VideoForest, HCI Initiative, HKUST 27

Thank you http://hci.cse.ust.hk/ Zhida Sun zhida.sun@connect.ust.hk