Multimedia Databases Storage and Retrieval of Media Data Jukka - PowerPoint PPT Presentation

Multimedia Databases Storage and Retrieval of Media Data Jukka Teuhola Dept. of Information Technology, University of Turku Fall 2012 MMDB-1 J. Teuhola 2012 1

General course info � http://staff.cs.utu.fi/kurssit/multimedia_databases/autumn_2012/ � Lectures 28 h, Tue 14-16, Fri 10-12, in classroom B2033 � Homework 6 times, every solution gives a bonus of 1 point to the final score in the examination. � Examination : � 5 tasks, max 8 points, so the total score is 0-40 points � Minimum accepted = 20 points, giving grade 1 � Linear interpolation: 20-40 � 1-5, formula: + points bonus - 20 = + grade round ( 1 ) 5 � Preliminary knowledge: Databases; data structures and algorithms MMDB-1 J. Teuhola 2012 2

Course material � Powerpoint slides : <course homepage>/slides Optional reading: � H. M. Blanken, A. P. de Vries, H. E. Blok, L. Feng (Eds.): Multimedia Retrieval , Springer 2007. � L. Dunckley: Multimedia Databases – An Object-Relational Approach , Addison-Wesley, 2003. � P. Rigaux, M. Scholl, A. Voisard: Spatial Databases, with Application to GIS , Morgan-Kaufmann, 2002. � D. C. Gibbon, Z. Liu: "Introduction to Video Search Engines", Springer 2008. � Miscellaneous articles. MMDB-1 J. Teuhola 2012 3

Course Contents (tentative) 1. Introduction 2. Management of Large Objects 3. Text and Document Databases 4. Multidimensional Data Structures 5. Spatial Databases 6. Image Databases 7. Video Databases 8. Audio Databases 9. Integration and Standardization MMDB-1 J. Teuhola 2012 4

Main themes of the course Structural and algorithmic database issues: � Storage principles � Data representation � Queries, searching, content-based retrieval � Indexing NOT: � Usage of software products � MM authoring and content production � MM presentation MMDB-1 J. Teuhola 2012 5

1. Introduction: ‘Multimedia revolution’ � What is multimedia? A dataset or document containing at least two different media types. � Multimedia and imaging are continuously growing trends. � Enhanced quality and quantity of information, compared to plain text � Brings dramatic improvements to human-computer interaction � Rich and expressive way of representing, browsing and interacting with information � “Second information revolution” � Revolutionizes business, science, engineering, manufacturing, art, entertainment… � Crucial issues: � Size can be huge � Speed required to satisfy audio/video transmission rates � Semantics : both type- and instance-level metadata MMDB-1 J. Teuhola 2012 6

Levels of sophistication in multimedia databases (1) MM files and archives: (3) MM archive as part of a wider application � Simple browsing and retrieval � Browsing/search by keyword, � No queries plus related actions, e.g. � Supporting software: e.g. web/ � Web shop: Amazon media server, browser, player (4) ’True’ MM databases: (2) Annotated & indexed archives: � General queries by media � Search by keyword; see e.g. content, e.g. � Image archive: Gimp-Savvy � Painting search: Hermitage � Audio archive: Spotify � Melody search: Musipedia � Spatial db: Google maps � Video archive: YouTube MMDB-1 J. Teuhola 2012 7

Multimedia data types ( a) Text � Integrated to most multimedia applications; complements (as metadata ) non-textual forms of data. � Nowadays text is usually structured/formatted by markup (e.g. XML) � Visual variability through fonts and layout � The most space-effective data type to store (b) Audio � Increasingly popular data type � Different formats (WAV, CD, MP3, AU, AIFF, QT, RA, WMA, Vorbis) � Digitized audio rather space-consuming (tens of Kbytes per second) � Compression is needed (e.g. MP3 compression ratio 12:1) � More compact: synthetic music in MIDI format (Musical Instrument Digital Interface); MPEG-4 SA (Structured Audio) MMDB-1 J. Teuhola 2012 8

Multimedia data types (cont.) (c) Still raster images � Black-and-white / grey-scale / color � One high-resolution image may take several megabytes � Large number of image formats (GIF, TIFF, JPEG, JP2, PNG, …) � Lossy compression ratio (e.g. for JPEG) normally about 1:10 (d) Vector graphics � 2D or 3D drawings, models, maps � Rather space-effective; consists of larger objects than pixels � Parameters of (meta) objects: scaling, orientation, rotation, etc. � Applications: CAD (computer-aided design), GIS (geographic information systems), animations, computer games (e) Integrated documents (text & images) � Can be generated by today’s text processing programs MMDB-1 J. Teuhola 2012 9

Multimedia data types (cont.) (f) Digital video Sequence of frames (= still images) � Highly data-intensive � Integrated audio (interleaved in playback time-sequencing) � � Higher compression ratio than with still images (subsequent frames resemble each other). Compression, transmission and decompression speed must be � 20-30 frames per second. Animations less space-intensive (synthetic images, standard shapes) � MPEG-4: Object-based representation, many special techniques � Streaming formats: ASF (MicroSoft), QuickTime (Apple), RM (RealMedia), � FLV (Flash video), WebM (Google) (g) General integrated multimedia/hypermedia presentations MS Powerpoint, Adobe Flash, SMIL � MMDB-1 J. Teuhola 2012 10

Sample application areas of MMDBs (a) Educational multimedia services: � Distance learning � Teaching material � Educational audio/video document archives � Preview possibility (b) Video-on-demand: � Selection of movie, possibly using queries � Preview possibility; wind/rewind � Requires high bandwidth � Method of payment must be simple MMDB-1 J. Teuhola 2012 11

Sample application areas (cont.) (c) Audio-on-demand: � Less bandwidth-consuming than video � Recorded programs, music, and live net radio stations (d) Electronic commerce : � Online info about products: pictures, explanations, availability, etc. � Possibility to make queries � Online ordering systems with credit card / net bank payment. � Examples: bookstore, travel agency (e) Intelligent systems (‘expert systems’) : � Machine repair: Automatic assistants of different repair jobs. Manuals may be hard to read; demonstrative videos tell it better � Medical care: Standard surgery operations � Crime investigations: combination of surveillance & other info MMDB-1 J. Teuhola 2012 12

Sample application areas of MMDBs (cont.) (f) Digital libraries � Organized collections of digital information � Both documents and their metadata in digital form � Versatile metadata- and content-based retrieval opportunities � Usually accessible through the web � The web itself & search engines may be considered some kind of (poorly organized) digital library (g) Medical information systems � Patient data, including X-rays, EKG curves, MRI images, ... � Strict confidentiality � Used for diagnosis, monitoring and research � Automated tools: image/signal processing, pattern recognition, ... MMDB-1 J. Teuhola 2012 13

General observation � All multimedia applications share some common aspects and functions. � The goal of this course is to find the domain-independent set of “core algorithms” which can be used in many applications by varying a few parameters. � A generalized multimedia DBMS (MMDBMS) would be useful; probably as an extension to a standard DBMS. MMDB-1 J. Teuhola 2012 14

Technology enabling multimedia � Hardware components : High-speed processors (CPU, GPU), high- performance multimedia workstations, scanners, digital cameras, video cameras, high-resolution monitors, touch-screen monitors, high-precision printers and plotters. � High-bandwidth networks (WAN, LAN, mobile), fiber optics, network standards � High-capacity storage devices : hard disks, optical disks and jukeboxes, solid-state & non-volatile memories. � Image/video processing software : Compression (JPEG, MPEG), analysis, filtering, segmenting, feature extraction. � CAD and animation software : 2D and 3D graphics, applications in science, engineering, medicine, computer games, etc. � Pattern recognition (characters, shapes, etc.): E.g. neural networks � Advanced software systems : OO languages, OO databases, operating systems, multithreading, etc. MMDB-1 J. Teuhola 2012 15

‘Definition’ of MMDB (1) Supports the main types of MM data (2) Can handle a very large number of MM objects (3) Supports high-performance, high-capacity storage management: Hierarchical storage (on-line, near-line, off-line) (4) Offers DB capabilities: Persistence, transactions, concurrency control, recovery from failures, querying with high-level declarative constructs, versioning, integrity constraints, security. (5) Information-retrieval capabilities: Exact-match retrieval, probabilistic (best-match) retrieval, content-based retrieval, ranking of results MMDB-1 J. Teuhola 2012 16