Felix Saurbier, Matthias Springstein Hamburg, November 6 SWIB 2017 - - PowerPoint PPT Presentation

▶
felix saurbier matthias springstein hamburg november 6
SMART_READER_LITE
LIVE PREVIEW

Felix Saurbier, Matthias Springstein Hamburg, November 6 SWIB 2017 - - PowerPoint PPT Presentation

Aug 13, 2023 250 likes •494 views

Visual Concept Detection and Linked Open Data at the TIB AV- Portal Felix Saurbier, Matthias Springstein Hamburg, November 6 SWIB 2017 Agenda 1. TIB and TIB AV-Portal 2. Automated Video Analysis 3. Visual Concept Detection 4. Data Quality

slide-1
SLIDE 1

Felix Saurbier, Matthias Springstein Hamburg, November 6 SWIB 2017

Visual Concept Detection and Linked Open Data at the TIB AV- Portal

slide-2
SLIDE 2

Agenda

  • 1. TIB and TIB AV-Portal
  • 2. Automated Video Analysis
  • 3. Visual Concept Detection
  • 4. Data Quality
  • 5. Data Model
  • 6. Data Publication & Reuse
slide-3
SLIDE 3

Agenda

  • 1. TIB and TIB AV-Portal
  • 2. Automated Video Analysis
  • 3. Visual Concept Detection
  • 4. Data Quality
  • 5. Data Model
  • 6. Data Publication & Reuse
slide-4
SLIDE 4

Page 4

Technische Informationsbibliothek (TIB)

  • German National Library of Science and Technology
  • University Library at Hannover
  • The world’s largest science and technology library
  • An infrastructure provider for the whole scientific work process
  • TIB strategy: Move beyond text
  • Competence Centre for Non-Textual Materials
  • Visual Analytics Research Group
slide-5
SLIDE 5

Page 5

  • Platform for quality-tested scientific videos
  • Online since April 2014
  • Developed by TIB and Hasso Plattner Institute

TIB AV-Portal (av.tib.eu)

  • 11,500 Videos (December 2017)
  • Conference recordings, lectures, experiments, video

abstracts, simulations, animations

  • Videos predominantly under open access licenses
  • Automatic metadata enrichment, DOI/MFID,

long-term preservation, semantic search

slide-6
SLIDE 6

Agenda

  • 1. TIB and TIB AV-Portal
  • 2. Automated Video Analysis
  • 3. Visual Concept Detection
  • 4. Data Quality
  • 5. Data Model
  • 6. Data Publication & Reuse
slide-7
SLIDE 7

Page 7

Scene Recognition (SBD) Speech Recognition (ASR) Image Recognition (VCD) Named Entity Linking (NEL) Text Recognition (OCR)

Video Analysis – Process

slide-8
SLIDE 8

Page 8

Video Analysis – Results

Video Segments Audio Transcript Named Entities

slide-9
SLIDE 9

Page 9

Video Analysis – Results (VCD)

Video Keyframes Visual Concepts

slide-10
SLIDE 10

Agenda

  • 1. TIB and TIB AV-Portal
  • 2. Automated Video Analysis
  • 3. Visual Concept Detection
  • 4. Data Quality
  • 5. Data Model
  • 6. Data Publication & Reuse
slide-11
SLIDE 11

Page 11

Visual Concept Detection – Supervised Learning

  • Supervised Learning Pipeline
  • Training: Modify the model parameters to reduce the classification loss
  • Prediction: Use the trained model to propagate the label of new data
slide-12
SLIDE 12

Page 12

Visual Concept Detection – Previous Approach

SIFT BoVW SVM

  • System is trained on a manually annotated dataset with over 8000 images
  • Classification of 49 visual concepts (16 deployed)
slide-13
SLIDE 13

Page 13

Visual Concept Detection – Current Approach

  • Utilizing a deep learning approach (Convolutional Neural Network)
  • Training feature extraction and classifier model together
slide-14
SLIDE 14

Page 14

  • Dataset
  • System is trained on a semi-supervised dataset with 50,000 images
  • Utilizing Google Image Search to find training samples
  • VCD Modul
  • Using Inception-Resnet-v2 network structure designed by Google
  • Neural network pre-trained with one million images
  • Classification of 73 visual concepts
  • Trained for 40 epochs

Visual Concept Detection – Current Approach

slide-15
SLIDE 15

Agenda

  • 1. TIB and TIB AV-Portal
  • 2. Automated Video Analysis
  • 3. Visual Concept Detection
  • 4. Data Quality
  • 5. Data Model
  • 6. Data Publication & Reuse
slide-16
SLIDE 16

Page 16

  • Validation during training
  • Using 1100 manually annotated images
  • Estimate the mean average precision for each concept

 0.33 mAP over all concepts

  • Compute the F1-Score to determine thresholds for the binary label
  • Testing
  • Separate testing for the whole processing pipeline
  • Future Work
  • Adjust the threshold
  • Filter noisy images in the training dataset

Data Quality

slide-17
SLIDE 17

Agenda

  • 1. TIB and TIB AV-Portal
  • 2. Automated Video Analysis
  • 3. Visual Concept Detection
  • 4. Data Quality
  • 5. Data Model
  • 6. Data Publication & Reuse
slide-18
SLIDE 18

Page 18

Data Model

Vocabularies

  • Bibframe Vocabulary
  • DCMI Metada Terms
  • DCMI Type Vocabulary
  • Friend of a Friend Vocabulary
  • Open Annotation Data Model
  • NLP Interchange Format
  • Internationalization Tag Set (ITS) Ontology

https://av.tib.eu/opendata

tib:vcd/15907_1291662_30904

  • a:hasTarget tib:video/15907#t=smpte-25:0:20:36:04 ;
  • a:annotatedBy tib:annotator/VCD-1.0.0 ;
  • a:hasBody tib:visualconcepts/molecular_geometry .

tib:visualconcepts/molecular_geometry skos:related gnd:4170383-2 .

Resource Description Framework (RDF)

slide-19
SLIDE 19

Page 19

Data Model

tib:video/15907 tib:video/15907#t=smpte-25:0:20:36:04 tib:vcd/15907_1291662_30904

dcterms:isPartOf

  • a:hasTarget
  • a:annotation

rdf:type

tib:annotator/VCD-1.0.0

  • a:annotatedBy

tib:visualconcepts/molecular_geometry

  • a:hasBody
  • a:semanticTag

rdf:type skos:related

gnd:4170383-2 wd:Q911331

slide-20
SLIDE 20

Agenda

  • 1. TIB and TIB AV-Portal
  • 2. Automated Video Analysis
  • 3. Visual Concept Detection
  • 4. Data Quality
  • 5. Data Model
  • 6. Data Publication & Reuse
slide-21
SLIDE 21

Page 21

  • CC0 RDF dumps
  • Dereferencable URIs & content negotiation with LodView
  • LDF server at https://labs.tib.eu/ldf
  • Planned: public SPARQL endpoint

Metadata Publication & Linked Open Data

slide-22
SLIDE 22

Page 22

  • Library catalogues & discovery services
  • Virtual libraries
  • Interlinking & Mash-Up

Reuse

slide-23
SLIDE 23

Contact Felix Saurbier T +49 511 762-14645, felix.saurbier@tib.eu

More Infos KNM@tib.eu av.tib.eu