Crossing Media for I m proved I nform ation Access the Reveal This - - PowerPoint PPT Presentation

LangTech, 29 February 20 08

Stelios Piperidis ILSP spip@ilsp.gr

Crossing Media for I m proved I nform ation Access

the Reveal This exam ple

2 LangTech, 29 February 20 08

• "The vision I have for the Web is about

anything being potentially connected w ith anything. It is a vision that provides us with new freedom, and allows us to grow faster than we ever could. . . . it brings the w orkings of society closer to the w orkings of our m inds."

Tim Berners-Lee : Weaving the Web, 2000

• “European citizens should be able to watch
• r listen to audiovisual content anytim e,

anyw here and on all technical platform s (TVset, computer, mobile phone, personal digital assistant, etc.)”

European Commission i2010 initiative

3 LangTech, 29 February 20 08

Vision

• Pers. Entertainment

System Laptop Local Repositor y Car Entertainment System E

• book

Reader PDA Radio TV / Satellite TV Home/Office PC Stereo Drama Personal Digital Images and Video Music Educational The Web /

• Dig. Libraries

Cinema News / TeleText Music/Voice Music/Voice Video/Text Video/Text/ Images Video/Text/ Ima ges/Music

• Pers. Entertainment

System

4 LangTech, 29 February 20 08

Multim edia Content Analysis Objectives

• develop content processing systems that help

people keep up with the explosion of digital content scattered over different platforms (radio, TV, World Wide Web), different media (speech, text, image, video) and different languages

• develop technology able to semantically index,

categorise, summarise and cross-link multiplatform, multimedia and multilingual digital content

5 LangTech, 29 February 20 08

A system that offers both types of service : a) Multimedia and Cross lingual Information Retrieval (pull) b) Multimedia and Cross lingual information Filtering (push)

Use Scenaria

Content Aggregator Media Archive

Search archive

WEB Radio

TV, Radio, Web data

TV

Multimedia technology

Mobile phone and Web interfaces

Local Archive User User profile Mobile Web

Delivery

6 LangTech, 29 February 20 08

Potential Users

• end users to gather, filter and categorize

information collected from a wide variety of sources in accordance with their preferences.

• professionals (media monitoring experts,

journalists and editors with demanding media retrieval needs – pull model)

• Laymen (novice technology users with

information collection/ consumption needs – push model)

• content providers

to add value to their content, restructure and re-purpose it and

• ffer their clients (subscribers, viewers, etc)

individual or corporate users, personalized content

7 LangTech, 29 February 20 08

Medium specific m etadata

text: terms/ keywords, named entities (e.g. names of persons, places, organizations), events and topics speech: speech/ nonspeech, speakers (e.g. speaker identity), transcriptions and stories video and im ages: keyframes and thematic categories, faces and persons

8 LangTech, 29 February 20 08

Cross-m ediality in m ultim edia analysis

News on Elections Source B Radio Broadcast News on Elections Single Source TV Broadcast First Interpretation Second Interpretation Audio (Speech/Music) Vidoe/Images Text Source A TV Broadcast

referring to different sources of information (radio and web text on sam e topic) → across docum ents referring to medium used to convey information within one source (audio, text, image of video segm ent) → within document

Video / I mages

9 LangTech, 29 February 20 08

Cross-m ediality in m ultim edia analysis

referring to medium used to convey information within one source (audio, text, image of video segm ent)→ within document Cross-media indexing

• treat imprecisions & inconsistencies
• process metadata of speech-image-text

Cross-media categorisation

• add to m etadata set
• process text and images

Cross-media summarisation

• add to m etadata set
• process video and text
• present video+ text+ audio salient parts

using a content/ domain specific multimedia discourse grammar

1 0 LangTech, 29 February 20 08

Cross-m ediality in m ultim edia analysis

referring to different sources of information (radio and web text on sam e topic) → across docum ents Semantic retrieval

• retrieval of different m ultimedia documents

for a specific query

• multidocument summarisation

1 1 LangTech, 29 February 20 08

audio Media Manager SPC - audio processing w eb radio tv cross- m edia stories Sm art Content story-based text/ im age/ video analysis categorisation, sum m arisation, translation text video FDI C - face analysis TPC - text processing text

XML m etadata speaker turns, speakers, text

keyfram e s XML m etadata

nam ed entities, term s, events XML m etadata

faces & ids XML Merging Segm ent Unification Story Boundary Detection Media Server I AC - video and im age processing

XML m etadata shotcuts, keyfram es, im age features

1 2 LangTech, 29 February 20 08

Exam ples of m ultim edia analysis m odules in a nutshell

www. reveal-this.

• rg

Retrieving stories Cross- lingual document translation Query Translation Cross- media summaries Scenes and visual summaries Textual summarisa tion in EL Text summarisa tion in EN Cross- media Categorisat ion Cross- media indexer Fact extraction in EL Fact extraction in EN Face detection & identificatio n Image Categorisat ion Speech Recognition in EL Speech recognition in EN

1 3 LangTech, 29 February 20 08

Cross-m edia sum m arisation architecture

Cross-media Summarization Subsystem

Analysis

TEXTUAL-BASED ENGINE

Analysis

SCENE Grouping CLUSTERING

Summarization Technologies Cross-lingual Translation Subsystem

REVEAL Translation Engine

Summarization Interfaces

Interfaces Summary Enrichment summary summary

Personalization Mechanisms

Users Profiles

1 4 LangTech, 29 February 20 08

Different dom ains: different m odels

Anchor Reportage Interview Reportage Interview Reportage Fight against Terrorism Arms Embargo … …. …. Human Rights

History Lifestyle Landscape History

1 5 LangTech, 29 February 20 08

Anchor Reportage Interview

Visual Summariser

Textual Summariser Textual Summariser

Scene Labeller

P1

A

P2

A

P5

I

P6

R

P4

R

P8

R

P3

R

P7

I

Audio Segmentor

TV News HTML+TIME

Presentation Layer

1 6 LangTech, 29 February 20 08

Euro-Parliam ent Sessions : m edia analysis EbS PLENARY 2005/04/27 15:00

Fight against Terrorism Arms Embargo … …. …. Human Rights

1 7 LangTech, 29 February 20 08

Euro- Parliam ent Sessions structure & content

T1 S1 T1 S2 T1 S4 T2 S1 T2 S4 T3 S1 T4 S5 T5 S1 T5 S2 T1 S3

Topic 1 Topic 2 Topic 3 Topic 4 Topic 5

S1 S2 S3 S4 S1 S4 S1 S2 S1 S5

Session Regions Speakers

1 8 LangTech, 29 February 20 08

Speaker Identifier T1 S1 T1 S2 T1 S4 T2 S1 T2 S4 T3 S1 T4 S5 T5 S1 T5 S2 T1 S3

Topic 1 Topic 2 Topic 3 Topic 4 Topic 5

S1 S2 S3 S4 S1 S4 S1 S2 S1 S5

Term Extractor

EbS Sessions HTML+TIME

Presentation Layer Textual Summariser

1 9 LangTech, 29 February 20 08

Travel docum entaries: m edia analysis

BestOfGreece-DG-EN - Chapter: Athens

History Lifestyle Landscape History

2 0 LangTech, 29 February 20 08

Travel docum entaries bits and bolts

P1

C1 C2

P2

C1

P3

C2

P4

C2

P5

C3

P6

C1

P7

C1

C1 C1 C2 C3 C2 C3 C1

Story Chapter Thematic Categories Regions

2 1 LangTech, 29 February 20 08

Audio Segmentor

Lifestyle

Land scape

History

P1

C1 C2

P2

C1

P3

C2

P4

C2

P5

C3

P6

C1

P7

C1 Image Categoriser

Textual Summariser

Travel Vocabulary

Travel Documentaries HTML+TIME

Presentation Layer

Clustering

2 2 LangTech, 29 February 20 08

• Elaborate crossing media techniques for multimedia

authoring and presentation

• Cross-media based indexing and retrieval of multimedia

content

• Cross-media analysis for better understanding of

communicated messages

• Cross-media methods in robotic and cognitive systems
• Cross-media techniques for better simulation of knowledge

and/ or language acquisition processes

Crossing m edia in the future