20/09/2018 Media Retrieval (2) Prepared by Ling Guan Jose Lay Paisarn Muneesawang Ning Zhang Rui Zhang 1 Outlines (revisited) • Introduction: – Intellectual Foundation of Multimedia Information Retrieval – Retrieval Models • Text Retrieval: – Database, Bibliographic, and Keyword Searches • Content ‐ based Retrieval: – Object ‐ matching and beyond • Indexing: – Inverted file • Searching: – Multimodality and Query adaptation 2 1
20/09/2018 Content ‐ based Retrieval (the conventional approaches) • Images, Audio, and Video • Caption Text • Multimodalities • Search methods and search engines • Result visualization/presentation Let us get start with a tour! 3 Examples of The MMR systems • Text ‐ based (and Content ‐ based) Querying – Images • Google’s Image Search • Yahoo Image Search (AltaVista’s Image Search) • TinEye Image Search (Toronto Local) – Audio • Yahoo Audio Search • Compaq’s SpeechBot (no longer exist) – Video and Multiple modalities • Facebook.com • Lycos Multimedia Search (evolving from a single search engine into a focused network of community and social sites) • FAST AllTheWeb Search • emediasearch.com (covering the Middle East region) 4 2
20/09/2018 Google’s image search: new feature 5 Google’s image search: new feature 6 3
20/09/2018 Google’s image search: new feature 7 Yahoo (Alta Vista) Image Search Yahoo Image Search 8 4
20/09/2018 TinEye, developed by Idée, Inc 9 Compaq’s SpeechBot (closed by HP) 10 5
20/09/2018 FAST’s AllTheWeb Search (bought out by Overture) 11 Qwiki (acquired by Yahoo!) • Traditional search result presentation grid ‐ based • It turns static information into dynamic experience 12 6
20/09/2018 MediaMill • Semantic video search engine – Cross browser – Galaxy browser – Sphere browser 13 3D Visualization/Presentation ‐ MIT Tangible HCI Lab 14 7
20/09/2018 Non ‐ general searching sites Image/Video Hosting sites Vertical Search Engine • Primarily function as • Focus on a specific segment of online contents sharing sites for on ‐ line • community. Academic: – IEEE Explore, Google Scholar • Indexed large database • Map: with yellow page info provide searching features – Google Maps and StreetView, for users. Bing Maps and StreetSide. • Image hosting • Food and Shopping – Flickr, Panoramio, Picassa – Yummly, Bing shopping, Yelp • Video hosting • Information – YouTube – Wikipedia 15 The Extended Text ‐ based Systems • Processing textual information to infer on audio ‐ visual content. – Often rely on the filename. • koala.jpg – Using meta information embedded in the web page. • <IMG SRC=“ryerson1234.gif" WIDTH="300" HEIGHT="60" ALT=“koala watching TV"> – Further processing of the container page. • The webpage is called: Trailer for Starwars episode I: Phantom the Menace, and in there you find a link to the phantom.mpg file. 16 8
20/09/2018 Speech Pre ‐ processing • In a speech retrieval engine, e.g., SpeechBot, a speech processing mechanism is applied to convert audio data into textual data, then conventional text ‐ based indexing and searching could be applied. 17 Touring the First Generation CBR Systems • Visual ‐ based Querying – University of California at Irvine’s MARS (Initially developed at Univ. of Illinois) – IBM’s QBIC 18 9
20/09/2018 MARS 19 QBIC at Hermitage Museum 20 10
20/09/2018 What did you observe? • What has been extended? • Which applications may be supported? • What is the constraint? 21 CBR: a brief history • In 1970s, image data are commonly archived independently and indexed using text ‐ based databases. – Indexing carries subjectivity of annotation. – Manual and expensive. • Into 1990s, efforts are taken to remove person indexing and to automate the mechanism. – Image data themselves are used as indexes. – Full automation is idealized. – Content ‐ Based Retrieval was born. 22 11
20/09/2018 CBR: a brief history ‐ 2 • The CBR was founded on the computer vision and pattern recognition idealism and their techniques. – CBR employs object ‐ matching mechanisms. – Earlier systems are characterized by Query by Example approach. – Features commonly used are: Color, Texture, and Shape. – Motion features in case of videos. 23 CBR: a brief history ‐ 3 • As Computer vision ideal is yet to achieve, CBR found itself to be constrained in a green ‐ house. – Many set ‐ off for sophisticated feature exploration and computational reasoning adventures. Deep NNs have provided a viable solution. – A strong call for abandoning the full ‐ automation idealism for returning to human interaction. – HCI techniques such as relevant feedbacks started to roll ‐ out. 24 12
20/09/2018 Between Yesterday & Tomorrow • An interesting question to ask today may well be: is the object ‐ matching idea all we want? – Typical current CBR query would mostly read: retrieve all documents that presents a blonde wearing blue dress driving a red porsche. – Now let us suggest some offsets: How about retrieve all paintings created using chiaroscuro technique? What differences are there between Christmas Cards and Birthday Cards? • Where do you want to go today? chiaroscuro – pictorial representation in terms of light and shade without regard to color 25 The Passage • Go Deeper. Enhancing what we have got. – Human ‐ centered computing. – Distributed Computing. – Multi ‐ modalities querying and processing. • Go Sidewise. Call for better multimedia understanding. – Efforts to find out which information and semantics are useful and how they could be derived and managed – Concept ‐ based (or semantic) retrieval. – Transcending the Object ‐ Matching Idealism. – Multimedia information mining. – Big data – Deep learning – … 26 13
20/09/2018 Content ‐ based Retrieval? Traditional Bibliographic Objectives Object Matching Keyword Searches 27 Roadmap Test 1: Tuesday, October 16, 1 hour Coverage of material: everything studied up to Lecture 5. The 1 st half of the class is normal teaching, and the 2 nd is test time. Open notes (paper copies only). You may consider bringing in a sheet of paper summarizing the contents of the course up to and including Lecture 5. Material taught on October 16 will not be included in Test 1. Office hours: Monday, October 15, 2:30 ‐ 4:30pm, ENG 315. Submit your project proposal electronically by Friday, October 12, 11:59pm. 28 14
Recommend
More recommend
