Media Retrieval (2) Prepared by Ling Guan Jose Lay Paisarn - - PDF document

▶
media retrieval 2
SMART_READER_LITE
LIVE PREVIEW

Media Retrieval (2) Prepared by Ling Guan Jose Lay Paisarn - - PDF document

Dec 27, 2022 254 likes •416 views

20/09/2018 Media Retrieval (2) Prepared by Ling Guan Jose Lay Paisarn Muneesawang Ning Zhang Rui Zhang 1 Outlines (revisited) Introduction: Intellectual Foundation of Multimedia Information Retrieval Retrieval Models Text

slide-1
SLIDE 1

20/09/2018 1

Media Retrieval (2)

Prepared by

Ling Guan Jose Lay Paisarn Muneesawang Ning Zhang Rui Zhang

1

Outlines (revisited)

  • Introduction:

– Intellectual Foundation of Multimedia Information Retrieval – Retrieval Models

  • Text Retrieval:

– Database, Bibliographic, and Keyword Searches

  • Content‐based Retrieval:

– Object‐matching and beyond

  • Indexing:

– Inverted file

  • Searching:

– Multimodality and Query adaptation

2

slide-2
SLIDE 2

20/09/2018 2

Content‐based Retrieval

(the conventional approaches)

  • Images, Audio, and Video
  • Caption Text
  • Multimodalities
  • Search methods and search engines
  • Result visualization/presentation

Let us get start with a tour!

3

Examples of The MMR systems

  • Text‐based (and Content‐based) Querying

– Images

  • Google’s Image Search
  • Yahoo Image Search (AltaVista’s Image Search)
  • TinEye Image Search (Toronto Local)

– Audio

  • Yahoo Audio Search
  • Compaq’s SpeechBot (no longer exist)

– Video and Multiple modalities

  • Facebook.com
  • Lycos Multimedia Search (evolving from a single search engine into a

focused network of community and social sites)

  • FAST AllTheWeb Search
  • emediasearch.com (covering the Middle East region)

4

slide-3
SLIDE 3

20/09/2018 3

Google’s image search: new feature

5

Google’s image search: new feature

6

slide-4
SLIDE 4

20/09/2018 4

Google’s image search: new feature

7

Yahoo (Alta Vista) Image Search

8

Yahoo Image Search

slide-5
SLIDE 5

20/09/2018 5

TinEye, developed by Idée, Inc

9

Compaq’s SpeechBot

(closed by HP)

10

slide-6
SLIDE 6

20/09/2018 6

FAST’s AllTheWeb Search

(bought out by Overture)

11

Qwiki

(acquired by Yahoo!)

  • Traditional search result presentation grid‐based
  • It turns static information into dynamic experience

12

slide-7
SLIDE 7

20/09/2018 7

MediaMill

  • Semantic video search engine

– Cross browser – Galaxy browser – Sphere browser

13

3D Visualization/Presentation

‐ MIT Tangible HCI Lab

14

slide-8
SLIDE 8

20/09/2018 8

Non‐general searching sites

Image/Video Hosting sites

  • Primarily function as

sharing sites for on‐line community.

  • Indexed large database

provide searching features for users.

  • Image hosting

– Flickr, Panoramio, Picassa

  • Video hosting

– YouTube

Vertical Search Engine

  • Focus on a specific segment of
  • nline contents
  • Academic:

– IEEE Explore, Google Scholar

  • Map: with yellow page info

– Google Maps and StreetView, Bing Maps and StreetSide.

  • Food and Shopping

– Yummly, Bing shopping, Yelp

  • Information

– Wikipedia

15

The Extended Text‐based Systems

  • Processing textual information to infer on audio‐

visual content.

– Often rely on the filename.

  • koala.jpg

– Using meta information embedded in the web page.

  • <IMG SRC=“ryerson1234.gif" WIDTH="300" HEIGHT="60"

ALT=“koala watching TV">

– Further processing of the container page.

  • The webpage is called: Trailer for Starwars episode I: Phantom the

Menace, and in there you find a link to the phantom.mpg file.

16

slide-9
SLIDE 9

20/09/2018 9

Speech Pre‐processing

  • In a speech retrieval engine, e.g., SpeechBot, a

speech processing mechanism is applied to convert audio data into textual data, then conventional text‐based indexing and searching could be applied.

17

Touring the First Generation CBR Systems

  • Visual‐based Querying

– University of California at Irvine’s MARS (Initially developed at Univ. of Illinois) – IBM’s QBIC

18

slide-10
SLIDE 10

20/09/2018 10

MARS

19

QBIC at Hermitage Museum

20

slide-11
SLIDE 11

20/09/2018 11

What did you observe?

  • What has been

extended?

  • Which applications may

be supported?

  • What is the constraint?

21

CBR: a brief history

  • In 1970s, image data are commonly archived

independently and indexed using text‐based databases.

– Indexing carries subjectivity of annotation. – Manual and expensive.

  • Into 1990s, efforts are taken to remove person

indexing and to automate the mechanism.

– Image data themselves are used as indexes. – Full automation is idealized. – Content‐Based Retrieval was born.

22

slide-12
SLIDE 12

20/09/2018 12

CBR: a brief history ‐2

  • The CBR was founded on the computer vision

and pattern recognition idealism and their techniques.

– CBR employs object‐matching mechanisms. – Earlier systems are characterized by Query by Example approach. – Features commonly used are: Color, Texture, and Shape. – Motion features in case of videos.

23

CBR: a brief history ‐3

  • As Computer vision ideal is yet to achieve, CBR found

itself to be constrained in a green‐house.

– Many set‐off for sophisticated feature exploration and computational reasoning adventures. Deep NNs have provided a viable solution. – A strong call for abandoning the full‐automation idealism for returning to human interaction. – HCI techniques such as relevant feedbacks started to roll‐

  • ut.

24

slide-13
SLIDE 13

20/09/2018 13

Between Yesterday & Tomorrow

  • An interesting question to ask today may well be: is

the object‐matching idea all we want?

– Typical current CBR query would mostly read: retrieve all documents that presents a blonde wearing blue dress driving a red porsche. – Now let us suggest some offsets: How about retrieve all paintings created using chiaroscuro technique? What differences are there between Christmas Cards and Birthday Cards?

  • Where do you want to go today?

chiaroscuro – pictorial representation in terms of light and shade without regard to color

25

The Passage

  • Go Deeper. Enhancing what we have got.

– Human‐centered computing. – Distributed Computing. – Multi‐modalities querying and processing.

  • Go Sidewise. Call for better multimedia understanding.

– Efforts to find out which information and semantics are useful and how they could be derived and managed – Concept‐based (or semantic) retrieval. – Transcending the Object‐Matching Idealism. – Multimedia information mining. – Big data – Deep learning – …

26

slide-14
SLIDE 14

20/09/2018 14

Content‐based Retrieval?

Traditional Bibliographic Objectives Keyword Searches Object Matching

27

Roadmap

 Test 1: Tuesday, October 16, 1 hour

  • Coverage of material: everything studied up to Lecture 5.
  • The 1st half of the class is normal teaching, and the 2nd is test time.
  • Open notes (paper copies only). You may consider bringing in a

sheet of paper summarizing the contents of the course up to and including Lecture 5.

  • Material taught on October 16 will not be included in Test 1.
  • Office hours: Monday, October 15, 2:30‐4:30pm, ENG 315.

 Submit your project proposal electronically by Friday, October 12, 11:59pm.

28