Media Indexing & Retrieval Media Indexing & Retrieval - - PDF document

20/09/2018 1

Media Indexing & Retrieval Media Indexing & Retrieval

Prepared by

Ling Guan Jose Lay Paisarn Muneesawang Ning Zhang Rui Zhang

1

Background

• More and more media are becoming available online. In the past

decade, we have seen the proliferation of online newspaper, net radio, web‐TV, net games, etc. etc.

• The size of the Internet is estimated to have exceeded 13,706,770

PB (Petabytes = 1,000 TB), while the worldwide disk storage capacity in 2018 is in the volume of 1450 EB (Exabyte = 1,000,000 TB).

• Newly created digital data in the year of 2011 is about 1800

Exabyte.

• Over 60% of the online data are of audio and/or visual content.
• Naturally the information contained in these massive data would

need to be accessible for them to be useful.

• Thus, Multimedia Information Retrieval

2

20/09/2018 2

Prominent Venues for Indexing and Retrieval

• IEEE International Conference on Multimedia and

Expo (ICME).

• ACM International Conference on Multimedia (ACM‐

MM).

• IEEE Symposium on Multimedia (ISM).
• ACM International Conference Multimedia

Information Retrieval (ACM‐MIR).

• IEEE International Conference on Multimedia

Information Processing and Retrieval (MIPR), inaugurating in 2018.

• The list is long…

3

Interesting Facts about ICME

• ICME 2005: 380 out of the 850 papers submitted are

related to media indexing and retrieval.

• ICME 2006 (Organized by Ryerson University in

Toronto): about 500 out the 1200 papers submitted are related to media indexing and retrieval.

• ICME 2005: 80% of the industrial exhibits are on

digital asset management, a principal application area of multimedia retrieval.

• ICME 2009, 30% of the papers are on multimedia

indexing and retrieval.

• ICME 2014, 40% of the papers are on indexing and

retrieval, and digital libraries.

4

20/09/2018 3

Objectives

• To understand multimedia information

retrieval from:

– the intellectual foundation of the subject. – the operational techniques of the processes.

5

Outlines

• Introduction:

– Intellectual Foundation of Multimedia Information Retrieval – Retrieval Models

• Text Retrieval:

– Database, Bibliographic, and Keyword Searches

• Content‐based Retrieval:

– Object‐matching and beyond

• Indexing:

– Inverted file

• Searching:

– Multimodality and Query adaptation

6

20/09/2018 4

Intellectual Foundation

• Multimedia, Information, and Retrieval

– Multimedia characteristics. – The difference between Data and Information. – Data Retrieval and Information Retrieval. – The traditional information retrieval (IR)

• bjectives.
• Multimedia Information Retrieval
• Multimedia Information Retrieval Objectives

7

Multimedia

• Multiple media sources
• Multimodality
• Coordination
• Interactivity

8

20/09/2018 5

Data and Information

Traditional bibliographical organization differentiates the notion of work as opposed to document.

• Work is:

– an intellectual creation. – the disembodied message, the information.

• Document is:

– the material embodiment of an intellectual creation, the data.

9

Data and Information Retrievals

• Information (or work) differs from data

(or document).

• Information Retrieval should not be

confused with Data Retrieval.

10

20/09/2018 6

Retrieval

A duo of Indexing and Searching primitives.

• Indexing (annotation) deals with:

– Finding representations for information in the document. – Organizing the representation to facilitate efficient search.

• Searching deals with:

– Capturing and presenting an information need. – Assessing relevance.

11

Traditional IR Objectives

• To enable finding a document of which:

– The title, the author, or the subject is known.

• To locate a collection of documents:

– By an author, a subject, a specific kind of literature.

• To facilitate the choice of a document based
• n:

– Its edition or literary association.

12

20/09/2018 7

Multimedia Information Retrieval

Multimedia IR can be presented in many ways:

• Extension View
• Functional View
• Modality View
• Topical View
• Other Views

13

Extension View

• Traditional Information

Retrieval techniques, e.g. bag of words, TD‐IDF, can be extended to deal with various data types as a document could take any form.

• Operationally it is being

challenged by content‐ based techniques.

14

20/09/2018 8

Functional View

• Indexing:

– Information Description – Descriptions Organization

• Searching:

– Query Description – Relevance Assessment

Practically, information is reduced to mere viewpoints (e.g. car show with a lady in the front).

Information Description Query Description Information Identification Information Identification Documents Information Needs INDEXING SEARCHING DB 15

Modality View

• Number of modalities

– Single Modality – Multiple Modalities

• Chronological

– Text‐based – Content‐based – Concept‐based (semantic) – Context‐based

Image Text Video Audio MM

16

20/09/2018 9

Multimedia Document Text Audio Video Keywords Meta Segment Region

Temporal Spatial Semantics Global Structure Semantic

Modality View (2)

• Multimedia Description

17

Topical View

Multimedia Information Retrieval Storage Technology and Distributed Processing Library and Information Science Data Structure and Algorithms Pattern Recognition and Computer Vision Psychology and Human Computer Interaction Artificial Intelligence & Machin Learning

18

20/09/2018 10

Other Views

• Multimedia IR could also be looked at from

many other perspectives:

– Push and Pull Directives – Structured and Unstructured Data – Real‐time and Offline Processing – Automatic and Manual Indexing – Compressed and Spatial Domain Descriptions – And many more.

19

Multimedia Information Retrieval Objectives

Traditional Bibliographic Objectives Keyword Searches Object Matching

20

20/09/2018 11

Flag Pole

End of Introduction.

21

Text Retrieval

Text retrieval could normally be classified into two categories:

• Structured

– Database – Bibliographic Systems

• Unstructured

– Full‐text or Keyword Search

Unstructured Structured

22

20/09/2018 12

Database Systems

• In a database system:

– relations and other attributes are formally constructed. – Search is limited by those attributes and their relations.

23

Bibliographic System

• Click the screen above to start the demo

24

20/09/2018 13

Bibliographic Objectives Restated

• To enable finding a document of which:

– The title, the author, or the subject is known.

• To locate a collection of documents:

– By an author, a subject, a specific kind of literature.

• To facilitate the choice of a document

based on:

– Its edition or literary association.

25

Bibliographic Operation

• Documents are indexed by using surrogates.
• A surrogate is a description of an information

viewpoint.

• Document databases are reduced to surrogate

databases.

• Search are performed on the surrogate

databases.

26

20/09/2018 14

Bibliographic Constraints

• The manual annotation for describing any

information viewpoint is a costly and slow process.

• Clearly it stands as the major handicap. To

reduce the cost and speed up the process:

– Standardization. Thus, interoperability. Distributing cost among institutions. – Automatic Indexing.

27

Full‐Text Retrieval

• Full‐text retrieval is the champion of the

automatic indexing efforts.

• It compromises the advanced features of

bibliographic system such as collocation with speed and significantly cheaper cost.

28

20/09/2018 15

Full‐text Retrieval

Click on the screen to start the demo. 29

Full‐text Retrieval: another example

30

20/09/2018 16

Full‐text Retrieval Principle

• Luhn’s Ideas:

– The frequency of word appearance in an article furnishes a useful measurement of word significance. – Words exceeding upper cut‐off are too common, while below lower cut‐

• ff are rare. Significant

words lie in between.

31

Full‐text Retrieval Characteristics

• makes no attempt to understand

“information” (work) by dealing only with data such as using the statistical measures.

• is essentially a “data retrieval” scheme.
• has been highly successful and forms the back

bone of the Web Search Engine.

32

20/09/2018 17

Full‐Text Indexing

• Inverted File Indexing:

– Consider a document: Content Retrieval using Energy Histogram with LF‐DCT Coefficient and Segment Grid. – A list of keywords between lower and upper cut‐

• ffs are extracted: Content, Retrieval, Energy,

Histogram, LF‐DCT, Coefficient, Segment, Grid. – The keywords are used to populate the inverted index file. …

33

Full‐Text Indexing

In an inverted index for 20 documents: ….

• Coefficient = {1, 4, 18}
• Histogram = {1, 20}

…

• Retrieval = {1, 3, 7, 9, 12, 15, 18, 20}

…

The numbers within a bracket refers to the document(s) in which a particular keyword is found.

34

20/09/2018 18

Full‐Text Searching

• Search is performed by looking up the

inverted index file for the query keywords.

• Advanced systems would also use term

frequency (tf.idf) and other measures such as set operator, proximity operator, thesaurus, feedback, etc.

35

Full‐Text Retrieval Constraints

• Require Natural Language Understanding

– Synonym and Near Synonym.

• To search for Multimedia Information Retrieval

Documents, following keywords may be used: Multimedia, Information, Data, Document, Retrieval, Access, Search.

– Semantic ambiguity

• e.g.: the word “play” has many meanings.

– False Drops

• e.g.: Sydney Opera House vs. a soap opera by Sydney

de Sousa presented at the International house.

36

20/09/2018 19

Extending Full‐Text Retrieval

• Boolean Set and Precedence

– Set operators: AND, OR, NOT, XOR, and ().

• (A and B) or C. But, if a query asked: A and B and C and D, would a

document containing A,B,C be considered similar?

• Proximity Operator

– A and B to be within a paragraph. C and D separated by less than 10 words.

• Ranking

– Ranking is used to resolve the problem above by incrementally listing relevance documents. – The emphasis was used to be on recall rather than precision (not useful in big data).

37

Extending Full‐Text Retrieval

• Building Natural Language Capability through

AI, including Computational Intelligence (CI):

– Knowledge based System – Fuzzy System – Neural Network Learning – Relevance Feedback

• Recent Success but Computationally Cost.

38

20/09/2018 20

Full‐Text Retrieval ++

• To regain some of the bibliographical features,

measures have been taken:

– Using Dublin Core: a minimum set of metadata such as title, author, etc. – Describing structure using XML instead of presentation

• nly by HTML.

– TREC – Text RTrieval Conference

• The scheme relies on share‐responsibility by shifting

the annotation (and cost) to the creator of the document.

39

Flag Pole

End of Text Retrieval.

40