VideoForest: Interactive Visual Summarization of Video Streams Based - - PowerPoint PPT Presentation

▶
videoforest interactive visual summarization of
SMART_READER_LITE
LIVE PREVIEW

VideoForest: Interactive Visual Summarization of Video Streams Based - - PowerPoint PPT Presentation

Jan 30, 2023 247 likes •533 views

VideoForest: Interactive Visual Summarization of Video Streams Based on Danmu Data Zhida Sun , Mingfei Sun , Nan Cao , Xiaojuan Ma Department of Computer Science and Engineering, Hong Kong University of Science and Technology

slide-1
SLIDE 1

VideoForest: Interactive Visual Summarization of Video Streams Based on Danmu Data

Zhida Sun†, Mingfei Sun†, Nan Cao‡, Xiaojuan Ma†

†Department of Computer Science and Engineering, Hong Kong University of Science and Technology ‡College of Design and Innovation, TongJi University

slide-2
SLIDE 2
  • Motivation
  • System
  • Architecture
  • System overview
  • Preprocessing
  • Analysis
  • Visualization
  • Evaluation
  • Conclusion

Outline

2/24/2017 VideoForest, HCI Initiative, HKUST 2

slide-3
SLIDE 3

Video Summarization on Youtube

2/24/2017 VideoForest, HCI Initiative, HKUST 3

Title Description Screenshot Summary Category Review

slide-4
SLIDE 4
  • Internal summarization techniques
  • Video, audio, linguistic features
  • External summarization techniques
  • Audience’s preferences and interests

Summarization of Video Streams

2/24/2017 VideoForest, HCI Initiative, HKUST 4

Video co-summarization [Chu et al. 2015] User preferences based [Kannan et al. 2013]

slide-5
SLIDE 5
  • Summarization representation
  • Matrix form [Lu and Grauman 2013]
  • Illustration: Visual storyline & Imagehive

Summarization of Video Streams

2/24/2017 VideoForest, HCI Initiative, HKUST 5

Visual storylines [Chen et al., 2012] Imagehive [Tan et al., 2011]

slide-6
SLIDE 6

Live Streaming Platforms

2/24/2017 VideoForest, HCI Initiative, HKUST 6

slide-7
SLIDE 7

Video Platforms with Danmu

2/24/2017 VideoForest, HCI Initiative, HKUST 7

Danmaku screen capture Danmu screen capture

slide-8
SLIDE 8

System Architecture

2/24/2017 VideoForest, HCI Initiative, HKUST 8

Goal: Video + Danmu + Interaction  Summarization

slide-9
SLIDE 9

Video Demo

2/24/2017 VideoForest, HCI Initiative, HKUST 9

slide-10
SLIDE 10

Processing

2/24/2017 VideoForest, HCI Initiative, HKUST 10

slide-11
SLIDE 11
  • Visual feature extraction
  • Key frames extraction from compressed video

Feature Extraction

2/24/2017 VideoForest, HCI Initiative, HKUST 11

Hierarchical structure of a video sequence [Mendi and Bayrak 2010]

slide-12
SLIDE 12
  • Textual feature extraction
  • syntactic and semantic analysis

Feature extraction

2/24/2017 VideoForest, HCI Initiative, HKUST 12

Similarity Sentiment Topic Function Symbol Pattern

slide-13
SLIDE 13

Analysis

2/24/2017 VideoForest, HCI Initiative, HKUST 13

slide-14
SLIDE 14

Meta-frame, Session and Scene Cluster

2/24/2017 VideoForest, HCI Initiative, HKUST 14

… Meta-frame

  • Visual attributes: image content
  • Textual attributes: audience’s reaction
slide-15
SLIDE 15

Meta-frame, Session and Scene Cluster

2/24/2017 VideoForest, HCI Initiative, HKUST 15

Session

  • Color palette of the entire session
  • Synoptic linguistic attributes of danmu

in the time range

slide-16
SLIDE 16

Meta-frame, Session and Scene Cluster

2/24/2017 VideoForest, HCI Initiative, HKUST 16

Scene cluster

  • Danmu commentary density
slide-17
SLIDE 17

Visualization

2/24/2017 VideoForest, HCI Initiative, HKUST 17

slide-18
SLIDE 18

System Interface Overview

2/24/2017 VideoForest, HCI Initiative, HKUST 18

slide-19
SLIDE 19

Interactions

2/24/2017 VideoForest, HCI Initiative, HKUST 19

View switching New session generating Smart summarization Scene preview Highlighting

slide-20
SLIDE 20

Visual Metaphor and Encoding

2/24/2017 VideoForest, HCI Initiative, HKUST 20

slide-21
SLIDE 21

Session Summary View

2/24/2017 VideoForest, HCI Initiative, HKUST 21

slide-22
SLIDE 22

Meta-frame aggregation

2/24/2017 VideoForest, HCI Initiative, HKUST 22

slide-23
SLIDE 23

Circle Image Packing

2/24/2017 VideoForest, HCI Initiative, HKUST 23

slide-24
SLIDE 24

Storyline threading

2/24/2017 VideoForest, HCI Initiative, HKUST 24

slide-25
SLIDE 25

Case study

2/24/2017 VideoForest, HCI Initiative, HKUST 25

  • 205 seconds long
  • 2052 danmu posts
  • 61 I-frames
  • 1804 P-frames
  • 61 sessions
  • 9 scenes
slide-26
SLIDE 26
  • Thirteen experts
  • Live demo
  • Interview & open discussion
  • Insights
  • Applications

User observation

2/24/2017 VideoForest, HCI Initiative, HKUST 26

slide-27
SLIDE 27
  • Conclusion
  • Future work
  • Long video stream scalability
  • Hierarchical aggregation
  • Advanced video analysis techniques

Future Work and Conclusion

2/24/2017 VideoForest, HCI Initiative, HKUST 27

slide-28
SLIDE 28

Thank you

http://hci.cse.ust.hk/

Zhida Sun zhida.sun@connect.ust.hk