Ontology for Multimedia Applications Hiranmay Ghosh TCS Innovation - - PowerPoint PPT Presentation

▶
ontology for multimedia applications
SMART_READER_LITE
LIVE PREVIEW

Ontology for Multimedia Applications Hiranmay Ghosh TCS Innovation - - PowerPoint PPT Presentation

May 08, 2023 284 likes •676 views

Ontology for Multimedia Applications Hiranmay Ghosh TCS Innovation Labs, Delhi Contributors Santanu Chaudhury IIT Delhi Anupama Mallik IIT Delhi Hiranmay Ghosh TCS / IIT Delhi ... and many other students 2 November 19, 2013 Tutorial:

slide-1
SLIDE 1

Ontology for Multimedia Applications

Hiranmay Ghosh TCS Innovation Labs, Delhi

slide-2
SLIDE 2

November 19, 2013 Tutorial: Web Intelligence 2013 2

Contributors

Santanu Chaudhury IIT Delhi Anupama Mallik IIT Delhi Hiranmay Ghosh TCS / IIT Delhi ... and many other students

slide-3
SLIDE 3

November 19, 2013 Tutorial: Web Intelligence 2013 3

Agenda

Part I

  • Introduction
  • Semantic Web and Ontology
  • Multimedia Content Processing
  • Ontology for Multimedia Data Interpretation

Part II

  • Multimedia Web Ontology Language
  • Application Examples
  • Distributed Multimedia Applications
  • Conclusion
slide-4
SLIDE 4

November 19, 2013 Tutorial: Web Intelligence 2013 4

Part I

slide-5
SLIDE 5

November 19, 2013 Tutorial: Web Intelligence 2013 5

Agenda

Part I

  • Introduction
  • Semantic Web and Ontology
  • Semantic Multimedia Content Processing
  • Ontology for Multimedia Data Interpretation

Part II

  • Multimedia Web Ontology Language
  • Application Examples
  • Distributed Multimedia Applications
  • Conclusion
slide-6
SLIDE 6

November 19, 2013 Tutorial: Web Intelligence 2013 6

Multimedia for infotainment

slide-7
SLIDE 7

November 19, 2013 Tutorial: Web Intelligence 2013 7

Some statistics [2012]

Source: pingdom.com

slide-8
SLIDE 8

November 19, 2013 Tutorial: Web Intelligence 2013 8

How do we

Deal effectively with the large volume

  • f distributed multimedia data?

Organize

Retrieve

Navigate

Correlate

slide-9
SLIDE 9

November 19, 2013 Tutorial: Web Intelligence 2013 9

News aggregation

  • Speech
  • Video
  • Overlay Text
  • Image
  • Text
  • Speech
  • Video
  • Overlay Text
  • Image
  • Text
  • TV Channels
  • Newspapers
  • Social Media
  • Maps
  • TV Channels
  • Newspapers
  • Social Media
  • Maps
  • Aggregate
  • Summarize
  • Present
  • Navigate
  • Aggregate
  • Summarize
  • Present
  • Navigate
slide-10
SLIDE 10

November 19, 2013 Tutorial: Web Intelligence 2013 10

Digital Heritage

  • Dance forms
  • Music genres
  • Instruments
  • Myth
  • Scripture
  • Artistes
  • Schools …
  • Dance forms
  • Music genres
  • Instruments
  • Myth
  • Scripture
  • Artistes
  • Schools …
  • Videos
  • Still images
  • Document images
  • Text
  • Videos
  • Still images
  • Document images
  • Text
  • Retrieve
  • Navigate
  • Retrieve
  • Navigate
slide-11
SLIDE 11

November 19, 2013 Tutorial: Web Intelligence 2013 11

Agenda

Part I

  • Introduction
  • Semantic Web and Ontology
  • Multimedia Content Processing
  • Ontology for Multimedia Data Interpretation

Part II

  • Multimedia Web Ontology Language
  • Application Examples
  • Distributed Multimedia Applications
  • Conclusion
slide-12
SLIDE 12

November 19, 2013 Tutorial: Web Intelligence 2013 12

The Semantic Web

  • Semantic data modeling

– Concepts represented through symbols – Relations between the concepts

  • Common reference for interpretation of data from multiple

sources

  • Layers for

– Syntactic compatibility (XML) – Semantic interoperability (RDF, OWL)

W3C Standards

slide-13
SLIDE 13

November 19, 2013 Tutorial: Web Intelligence 2013 13

Ontology

  • A formal representation of a domain
  • An artiste is a person
  • A person has name [string]
  • A dancer is an artiste
  • A dancer performs dance
  • DancerX is a dancer
  • Bharatnatyam is a Dance
  • DancerX performs Bharatnatyam
  • DancerX has name “Yamini Krishnamurthy”

Dance Bharat- Natyam Artiste DancerX Dancer Person

Performs Performs String Has name “Yamini Krishnamurthy” Has name

slide-14
SLIDE 14

November 19, 2013 Tutorial: Web Intelligence 2013 14

Why use ontology?

  • Template for information extraction
  • Reasoning to find new facts (not explicitly stated)
  • Separation of knowledge from program logic facilitates

– Knowledge Engineering – Reuse and maintenance

<dancer><name><dance-type>

  • DancerX is a person
  • DancerX performs Dance
  • At least one dancer performs Bharatnatyam
slide-15
SLIDE 15

November 19, 2013 Tutorial: Web Intelligence 2013 15

Agenda

Part I

  • Introduction
  • Semantic Web and Ontology
  • Multimedia Content Processing
  • Ontology for Multimedia Data Interpretation

Part II

  • Multimedia Web Ontology Language
  • Application Examples
  • Distributed Multimedia Applications
  • Conclusion
slide-16
SLIDE 16

November 19, 2013 Tutorial: Web Intelligence 2013 16

Content, Concept & Context

  • Content based retrieval (early 1990's)

– Low level image features, e.g. Color & texture

  • Concept based (late 1990's – still evolving)

– Features conveying more semantics, e.g. SIFT – Machine Learning techniques

  • Contextual reasoning
  • Granularity of semantics

– Scene recognition – Object recognition

  • Generic & Specific

Water (Blue) Sky (Blue) Sand (Brown)

A beach scene

slide-17
SLIDE 17

November 19, 2013 Tutorial: Web Intelligence 2013 17

Current state of content understanding

  • Significant progress in visual data understanding

– Document images, Surveillance, Medical / Satellite imagery,

Scene understanding, Action recognition, ...

  • Audio & Speech

– Good progress

  • Domain specific solutions

– Implicit domain knowledge

slide-18
SLIDE 18

November 19, 2013 Tutorial: Web Intelligence 2013 18

Semantic gap: still an unsolved problem

Bananas J

  • y

a n d f r e e d

  • m

Bharatnatyam

Semantic Gap

Media World Semantic World S T OP ! Red Light

slide-19
SLIDE 19

November 19, 2013 Tutorial: Web Intelligence 2013 19

Agenda

Part I

  • Introduction
  • Semantic Web and Ontology
  • Multimedia Content Processing
  • Ontology for Multimedia Data Interpretation

Part II

  • Multimedia Web Ontology Language
  • Application Examples
  • Distributed Multimedia Applications
  • Conclusion
slide-20
SLIDE 20

November 19, 2013 Tutorial: Web Intelligence 2013 20

Multimedia Data Integration

  • Different Media types
  • Diversity in descriptors
  • Difference in indexing schemes

Can we do semantic modeling of multimedia data?

slide-21
SLIDE 21

November 19, 2013 Tutorial: Web Intelligence 2013 21

Working with the annotations

  • Multimedia data is often associated with annotation

– Structured metadata, User tags, HTML <ALT> tag, surrounding

text, ...

  • We can use ontology to interpret them?
  • A set of collaborating museums

– Well-curated media archives – Controlled metadata associated with media artifacts

  • OWL-based domain ontology for information integration
  • Unfortunately, it does not work with any arbitrary media collection

CIDOC: Early 2000's

slide-22
SLIDE 22

November 19, 2013 Tutorial: Web Intelligence 2013 22

Crowd-sourced data and knowledge

  • Semantics extracted

– From Crowd-sourced tags – With Crowd-sourced knowledge (Wikipedia) – A new line of research

  • But ...

– Estimated 70% of social media contents are without tags – Automatic tagging

(2008 onwards)

slide-23
SLIDE 23

November 19, 2013 Tutorial: Web Intelligence 2013 23

“Qualities” of concepts

  • “Qualities”: perceptible/measurable

–

Physical (color, size …)

–

Relations (Spatial and temporal)

  • Relation between and quality regions (qualia)

–

“Red” is opposite to “green”

–

“Red” is close to “brown”

Different shades of red Source: Gangemi (2002)

slide-24
SLIDE 24

November 19, 2013 Tutorial: Web Intelligence 2013 24

Multimedia Content Description Scheme

  • Flexible language to describe multimedia contents

– Representations (tools) for common audio and visual

features

  • Color, texture, shape, frequency spectrum, etc.

– Scene description

  • Structural and semantic description

– Extensible

  • Possible to define new descriptors

ISO Standard: MPEG-7: Early 2000's

slide-25
SLIDE 25

November 19, 2013 Tutorial: Web Intelligence 2013 25

Description of still image

slide-26
SLIDE 26

November 19, 2013 Tutorial: Web Intelligence 2013 26

Video segments

Segment-Relationship Graph Video segments and regions

slide-27
SLIDE 27

November 19, 2013 Tutorial: Web Intelligence 2013 27

Comments on MPEG-7

  • Accomplishes syntactic interoperability for multimedia
  • Describes multimedia document content

– XML based schema – Lots of flexibility (same scene can be described in many

different ways)

– No semantics, no support for reasoning

  • Quite a few MM Information system built with MPEG-7

– Template matching (query by example paradigm) –

slide-28
SLIDE 28

November 19, 2013 Tutorial: Web Intelligence 2013 28

Ontology for multimedia “concepts”

  • Controlled vocabulary for MPEG-7 semantic description

– Utility – Coverage – Feasibility – Observability

IBM + CMU Mid 2000's

Source: Naphade (2006)

slide-29
SLIDE 29

November 19, 2013 Tutorial: Web Intelligence 2013 29

MPEG-7 Ontologies

  • To provide semantic rigor to MPEG-7 descriptors
  • Several research projects

– Harmony – AceMedia – DS-MIRF – COMM – Boemie – ...

  • Converts MPEG-7 constructs to RDF / OWL constructs
  • Different coverage to MPEG-7 parts

Early-Mid 2000's

slide-30
SLIDE 30

November 19, 2013 Tutorial: Web Intelligence 2013 30

MPEG-7 Ontology: Class hierarchies

Multimedia Contents Image Multimedia Contents Video Still region Video segment Segment Still region Image text Mosaic Moving region Video text ... ... ... ... Audio Audio segment Top level content entities Video segment Segment classes

slide-31
SLIDE 31

November 19, 2013 Tutorial: Web Intelligence 2013 31

MPEG-7 Ontology: Media Properties

Visual Descriptor Color

subclass of

Still region Vedio segment

Domain MPEG-7 Color Tools Range Color Layout Scalable Color Dominant Color subclass of

slide-32
SLIDE 32

November 19, 2013 Tutorial: Web Intelligence 2013 32

MPEG-7 Ontologies ... contd.

  • Creates semantic description of multimedia contents in collections

– Excludes semantic descriptors

  • Integrates with domain ontology

– Usually with a core ontology

  • Examples: Harmony, AceMedia, COMM

– Includes semantic descriptors

  • Results in independent semantic descriptions of repositories
  • Needs common understanding of domain
  • Example: DS-MIRF
slide-33
SLIDE 33

November 19, 2013 Tutorial: Web Intelligence 2013 33

Interoperability models

MPEG-7 MPEG-7 Ontology

  • DS-MIRF

MPEG-7 Ontology

  • Harmony
  • AceMedia
  • COMM

Source: Dasiopoulou (2010)

slide-34
SLIDE 34

November 19, 2013 Tutorial: Web Intelligence 2013 34

Architecture for semantic integration

Speech Image Video Video OCR Text

Media Processing Concept Interpretation and Fusion Multimedia Ontology

Domain knowledge Media Models

slide-35
SLIDE 35

November 19, 2013 Tutorial: Web Intelligence 2013 35

Comments on MPEG-7 ontologies

  • Content model for documents / collections

– Can correlate diverse media forms

  • Specific to multimedia instances

– Not a generic collection independent ontology

  • Media model and domain model form separate layers

– Media interpretation does not benefit from domain

knowledge

slide-36
SLIDE 36

November 19, 2013 Tutorial: Web Intelligence 2013 36

Pictorially enhanced ontology

Univ Fierenze: Mid-Late 2000's

  • Visual templates (examples) are associated with

media events (concepts)

– Each template represents a distinct modality of

manifestation

  • New instances are classified based on feature

similarity with prototypes

– Automatic event detection and annotation

  • Domain ontology relates such events
slide-37
SLIDE 37

November 19, 2013 Tutorial: Web Intelligence 2013 37

Pictorially enhanced ontology ... contd

Source: Ballad, et al (2009)

slide-38
SLIDE 38

November 19, 2013 Tutorial: Web Intelligence 2013 38

End of Part I