VK Multimedia Information Systems Mathias Lux, - - PowerPoint PPT Presentation

▶
vk multimedia information systems
SMART_READER_LITE
LIVE PREVIEW

VK Multimedia Information Systems Mathias Lux, - - PowerPoint PPT Presentation

Jul 20, 2023 198 likes •1.08k views

VK Multimedia Information Systems Mathias Lux, mlux@itec.uni-klu.ac.at Dienstags, 16.oo Uhr c.t., E.2.69 This work is licensed under a Creative Commons Attribution-NonCommercial- ShareAlike 2.0 License. See

slide-1
SLIDE 1

Department for Information Technology, Klagenfurt University, Austria

VK Multimedia Information Systems

Mathias Lux, mlux@itec.uni-klu.ac.at Dienstags, 16.oo Uhr c.t., E.2.69

This work is licensed under a Creative Commons Attribution-NonCommercial- ShareAlike 2.0 License. See http://creativecommons.org/licenses/by-nc-sa/2.0/at/

slide-2
SLIDE 2

http://www.uni-klu.ac.at

2

Präsentationen 03.06.

  • Video Summary - Pörtsch
  • Flickr Related Tags – Pitman
  • MP3 LSA – Waltl & Jonke
  • YouTube Usage – Bartha

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

slide-3
SLIDE 3

http://www.uni-klu.ac.at

3

Präsentationen 24.06.

  • LSH - Kofler & Pasterk
  • Image Features – Finke & Unterzaucher
  • Seam Carving – Wanschou
  • Retrieval von Intentionen – Doliner &

Irrasch

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

slide-4
SLIDE 4

http://www.uni-klu.ac.at

4

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Contents

  • MPEG-7
  • MPEG-21
  • Metadata Generation & Annotation
  • Social Software & Metadata
slide-5
SLIDE 5

http://www.uni-klu.ac.at

5

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

MPEG-7

  • ISO/IEC Standard: Multimedia Content

Description Interface

  • Moving Pictures Expert Group

 Specification goes on ...

  • It’s based on XML (Schema)

 Binary representations possible (BiM)

  • Allows differing granularity of descriptions

 Extensive to very simple

slide-6
SLIDE 6

http://www.uni-klu.ac.at

6

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

MPEG-7 History

  • Call for Proposals: October 1998
  • Evaluation: February 1999
  • First version of Working Draft (WD): December

1999

  • Committee Draft (CD): October 2000
  • Final Committee Draft (FCD): February 2001
  • Final Draft International Standard (FDIS): July

2001

  • International Standard (IS): September 2001
slide-7
SLIDE 7

http://www.uni-klu.ac.at

7

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

MPEG-7 Basics

  • Descriptors

 Syntax and semantics of exactly one (low or high level) elementary feature  Also base data types are defined

  • Description Schemes

 Defines structures within a framework

  • Description Definition

Language (DDL)

 Extension of XML Schemes

  • Coding Schemes

 Create and interpret descriptions in BiM

slide-8
SLIDE 8

http://www.uni-klu.ac.at

8

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

MPEG-7 Parts

  • 1. MPEG-7 Systems
  • Tools needed to prepare MPEG-7 descriptions for efficient

transport and storage and the terminal architecture.

  • 2. MPEG-7 Description Definition Language
  • Language for defining the syntax of the MPEG-7 Description

Tools and for defining new Description Schemes.

  • 3. MPEG-7 Visual
  • Description Tools dealing with (only) visual descriptions.
  • 4. MPEG-7 Audio
  • Description Tools dealing with (only) audio descriptions.
  • 5. MPEG-7 Multimedia Description Schemes
  • Description Tools dealing with generic features and multimedia

descriptions.

slide-9
SLIDE 9

http://www.uni-klu.ac.at

9

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

MPEG-7 Parts

6. MPEG-7 Reference Software

  • Implementation of relevant parts of the MPEG-7 Standard with

normative status.

7. MPEG-7 Conformance Testing

  • Guidelines and procedures for testing conformance of MPEG-7

implementations

8. MPEG-7 Extraction and Use of Descriptions

  • Informative material about the extraction and use of some of

the Description Tools.

9. MPEG-7 Profiles and levels

  • Provides guidelines and standard profiles.
  • 10. MPEG-7 Schema Definition
  • Specifies the schema using the Description Definition

Language

slide-10
SLIDE 10

http://www.uni-klu.ac.at

10

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Scope of MPEG-7

from: http://www.chiariglione.org/mpeg/standards/mpeg-7/mpeg-7.htm

slide-11
SLIDE 11

http://www.uni-klu.ac.at

11

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Basic Elements

Basic elements are fundamental constructs and used throughout the whole MPEG-7 description

  • Basic datatypes

 Time and date, relative and absolute  Numeric datatypes like matrices and vectors

  • Links & Media Localization

 Interconnections and content linking

slide-12
SLIDE 12

http://www.uni-klu.ac.at

12

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Navigation & Access

  • Descriptors for Browsing & Retrieval

 Summaries  Partitions (time, space & frequency)  Decompositions (time, space & frequency)  Variations

slide-13
SLIDE 13

http://www.uni-klu.ac.at

13

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

User Interaction

  • Pertaining consumption of AV data

 user preferences  usage history

  • Meant to facilitate personalization

 Matching User Interaction DS with content description  Is research topic @ ITEC

slide-14
SLIDE 14

http://www.uni-klu.ac.at

14

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Content Organization

  • Organization & modelling of collections

 Audio-visual content, segments, events, and/or

  • bjects
  • E.g. pictures, scenes, music files, etc.

 Allows collection description as a whole

  • E.g. “Pictures of my holiday in Ebonia”
slide-15
SLIDE 15

http://www.uni-klu.ac.at

15

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Content Management

  • Creation & Classification

 Title, textual annotation, creators, creation locations, and dates.  Categories such as genre, subject, purpose or language.  Review and guidance information: Age classification, parental guidance, and subjective review.  Related material information.

  • Media coding, storage & file formats

 Media profiles & master media

  • Content Usage

 Usage rights, usage record, and financial information

slide-16
SLIDE 16

http://www.uni-klu.ac.at

16

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Content Description: Structural vs. Conceptual Aspects

  • Program DS (in sense of TV program)
  • Analogy to

 Table of content – Region tree (linear partitioning)  Index – Object tree (non-linear structure)

Table of Contents Chapter 1 ............. Section 1.1.... Section 1.2.... Chapter 2 ............. Section 2.1.... Section 2.2.... Index Topic 1 ............. Item 1.…… Item 2.…… Topic 2 ............. Item 1.……. Item 2.…….

slide-17
SLIDE 17

http://www.uni-klu.ac.at

17

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Content Description: Structural Aspects

  • Divide a video stream into physical and logical

video segments

  • The higher the

level of a physical video unit, the more semantic information is necessary

  • Logical units are

based on semantic content

slide-18
SLIDE 18

http://www.uni-klu.ac.at

18

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Region and Object Trees

slide-19
SLIDE 19

http://www.uni-klu.ac.at

19

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Content Description: Semantic Aspects

  • Low Level Features

 Extraction from Content  Descriptors for

  • Shape, color, texture (visual)
  • Timbre, rhythm (audio)
  • High Level Features

 Annotation  So called semantic descriptors

  • Textual information
  • Conceptual information
slide-20
SLIDE 20

http://www.uni-klu.ac.at

20

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

MPEG-7 High Level Descriptors

  • Textual Descriptions

 Text to describe temporal / spatial regions

  • The W’s

 Structured way of textual descriptions

  • Who, Where, What Object, When, Why, How & Where
  • Instead of textual descriptions

 Controlled Terms

  • Dictionaries, Taxonomies, Classifications Schemes

 Semantic Description Scheme

slide-21
SLIDE 21

http://www.uni-klu.ac.at

21

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

MPEG-7 Semantic Description Scheme

slide-22
SLIDE 22

http://www.uni-klu.ac.at

22

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Actual Description in MPEG-7

slide-23
SLIDE 23

http://www.uni-klu.ac.at

23

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Contents

  • MPEG-7
  • MPEG-21
  • Metadata Generation & Annotation
  • Social Software & Metadata
slide-24
SLIDE 24

http://www.uni-klu.ac.at

24

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

MPEG-21 – motivation and scope

slide-25
SLIDE 25

http://www.uni-klu.ac.at

25

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

MPEG-21 Objectives

MPEG-21’s goal is to create an interoperable and integrated multimedia framework in three steps:

  • 1. Develop “big picture”: understand how the

components of the framework are related and identify where gaps in the framework exist

  • 2. Fill the gaps: develop new standard

specifications where needed

  • 3. Integrate: achieve the integration of standards

to support harmonized technologies for the management of multimedia content

slide-26
SLIDE 26

http://www.uni-klu.ac.at

26

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

MPEG-21 Digital Item

A Digital Item (DI) is a structured digital

  • bject with a standard representation,

identification, and metadata within the MPEG-21 framework

  • Digital Items are “the content”
  • DIs consist of

 Resources (individual assets, distributed content),  Metadata (data about or pertaining the DI) and  Structure (relationships between parts of the DI)

slide-27
SLIDE 27

http://www.uni-klu.ac.at

27

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Digital Item - Example

The DI is the fundamental unit for distribution and transaction within the MPEG-21 framework.

slide-28
SLIDE 28

http://www.uni-klu.ac.at

28

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

MPEG-21 User and User Interaction

  • Any entity that interacts in the MPEG-21 environment or

makes use of a Digital Item

  • Users include individuals, organisations, corporations,

consortia, governments, other standards bodies, etc.

  • Roles including creators, consumers, rights holders, content

providers, distributors, etc.

  • Each User will assume specific rights and responsibilities

according to their interaction with other users

User A User B

Transaction/Use/Relationship Content Authorization/Value

Exchange

slide-29
SLIDE 29

http://www.uni-klu.ac.at

29

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Seven Architectural “Elements”

Vision Vision, , De Declaration, claration, an and Id d Identifi entifica cation tion

Digit igital al Righ ights ts Ma Managem agement ent Adap Adaptation ation Process Processin ing Systems Systems Misc Misc

  • Pt. 4: IPMP

Components

  • Pt. 5: Rights

Expression Lang

  • Pt. 6: Rights

Data Dictionary

  • Pt. 7: Digital

Item A daptation

  • Pt. 10: Digital

Item Processing Amd.1: C

  • nvers.

And Permissions Amd.2: Dynamic and Distributed Adaptation

  • Pt. 1: Vision, Technologies

and Strategy

  • Pt. 2: Digital Item

Declaration

  • Pt. 3: Digital Item

Identification

  • Pt. 9: File

Format

  • Pt. 16: Binary

Format

  • Pt. 18: Digital

Item Streaming

  • Pt. 8: Reference

Software

  • Pt. 11: Persistent

Association

  • Pt. 12: Test Bed
  • Pt. 14: Conform.
  • Pt. 15: Event

Reporting

  • Pt. 17: Fragment

Idenfication Amd.1: Add‘l C++ bindings Amd.1: DII relationship types

slide-30
SLIDE 30

http://www.uni-klu.ac.at

30

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

User A User B

Transaction/Use/Relationship

Digital Item

Authorization/Value Exchange 

Terminals & Networks Content Management and usage Intellectual Property management and Protection D ig ital I tem Iden tification andDescription D ig ital Ite m D eclaration :

  • Examples

Natural and Synthetic Scalability C

  • ntent

Representation Examples :

  • Storage Management
  • Content Personalisation

Examples :

  • Unique Identifiers
  • Content Descriptors

:

  • Examples

Resource Abstraction Resour ce Mgt. (QoS) Examples :

  • Encryption
  • Authentication
  • Watermarking

Event Reporting Examples :

  • “Container”
  • “Item”
  • “Resource”

Roles of the Architectural Elements

slide-31
SLIDE 31

http://www.uni-klu.ac.at

31

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Contents

  • MPEG-7
  • MPEG-21
  • Metadata Generation & Annotation
  • Social Software & Metadata
slide-32
SLIDE 32

http://www.uni-klu.ac.at

32

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Metadata Generation & Annotation

  • Process of creating data about data
  • Content has to be known

 Watch & understand video / image collection  Listen and assess audio

  • Metadata standard has to be known

 What are the possible fields  What are the used classification systems

slide-33
SLIDE 33

http://www.uni-klu.ac.at

33

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Evaluation (1/2)

  • Goal: Identify the opinion of users on

manual semantic annotation

  • 5 Users with following (median)

background:

 17 years of computer experience  Using a computer 50 h / week  2 years experience with digital photo cameras  4 years experience with imaging software

slide-34
SLIDE 34

http://www.uni-klu.ac.at

34

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Evaluation (2/2)

  • 2 Tasks were given:

 Annotate a photo with a given description and an extensive prior introduction to semantic photo annotation with Caliph

  • video was shown,
  • concept was explained and
  • questions were answered

 Annotate a photo fully on your own  After Tasks:

  • Questionnaire with several subjective questions
  • Evaluation Scale from: -3 (strongly disagree) to 3

(strongly agree)

slide-35
SLIDE 35

http://www.uni-klu.ac.at

35

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Evaluation Results: General Questions

 The concept of meta data is very new to me: -2.6  It was easy to understand the concept of semantic meta data while using Caliph: 1.8  The visualization of the semantic meta data within Caliph is easy to understand and interpret: 2.2  The annotation of images with textual descriptions can be done fast and easily: 1.4  The annotation of images with semantic meta data can be done fast and easily: 1.2  I can see an obvious benefit by using semantic meta data for image (multimedia) annotation: 1.4

Scale: (disagree) -3 to 3 (agree)

slide-36
SLIDE 36

http://www.uni-klu.ac.at

36

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Evaluation Results: Scenario based questions

  • 1. The complexity of semantic

annotation is too high to be useful for organizing photos.

  • 2. I would find it easy to

annotate a large set digital photos (e.g. 100+).

  • 3. I would recommend Caliph or

a similar tool to annotate digital photos.

  • 4. I can see an obvious benefit

by using semantic meta data for the organization of photos.

Personal Newspaper

  • 0.6
  • 1.8
  • 0.6
  • 0.2

0.8 1.4 1.4 2.2

Scale: (disagree) -3 to 3 (agree)

slide-37
SLIDE 37

http://www.uni-klu.ac.at

37

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Evaluation Results: Annotation performance

5 10 15 20 25 User 1 User 2 User 3 User 4 User 5 min. Test 1 Test 2

slide-38
SLIDE 38

http://www.uni-klu.ac.at

38

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Evaluation Results: Annotation performance

 Median times for annotation:

  • 15.4 minutes for the 1st test and
  • 6 minutes for the 2nd test

 Median time in a self test with 17 photos:

  • 1 minute and 53 seconds per photo

 That results in an approximate time of 10 h 27

  • min. for annotation of a set of 333 photos
slide-39
SLIDE 39

http://www.uni-klu.ac.at

39

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Evaluation Results: Diversity

  • f Annotations (2nd test)
  • Structured text annotation field “Who”:
  • 1. Vedran, Wolfgang, Armin
  • 2. Wolf, Armin, Vedran
  • 3. Wolfgang Kienreich, Vedran Sabol, Armin Ulbrich
  • 4. wolfgang, armin, vedran
  • 5. W.Kienreich,A.Ulbrich,V.Sabol
slide-40
SLIDE 40

http://www.uni-klu.ac.at

40

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Evaluation Results: Diversity

  • f Annotations (2nd test)
  • Free text annotation:

1. Stadthalle, Graz, Austria I-Know '04 Knowledge Managment Conference 2. The three are sitting ... 3. Wolfgang Kienreich, Armin Ulbrich und Vedran Sabol (v.l.n.r.) sprechen miteinander auf der I-Know.Wolfgang Kienreich, Vedran Sabol, Armin Ulbrich are at I-Know, Graz for Talking 4. Stadthalle, Graz, Austria I-Know '04 Knowledge Managment Conference 5. Wolfgang,Armin and Vedran talking to each other on I-Know 04 at Stadthalle Graz.

slide-41
SLIDE 41

http://www.uni-klu.ac.at

41

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Evaluation Results: Diversity

  • f Annotations (2nd test)

User 1: User 2:

slide-42
SLIDE 42

http://www.uni-klu.ac.at

42

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Evaluation Results: Diversity

  • f Annotations (2nd test)

User 3: User 4:

slide-43
SLIDE 43

http://www.uni-klu.ac.at

43

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Evaluation Results: Diversity

  • f Annotations (2nd test)

User 5:

slide-44
SLIDE 44

http://www.uni-klu.ac.at

44

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Lessons Learned

 Users like the graphical annotations editor  Users see semantic annotation in a professional (business) environment  Semantic annotation is very time consuming  The MPEG-7 nomenclature is not intuitive

  • Semantic agent / place / object & relations
  • Creator of image / description / quality rating

 Tagging with central tag repository …

slide-45
SLIDE 45

http://www.uni-klu.ac.at

45

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Demo

slide-46
SLIDE 46

http://www.uni-klu.ac.at

46

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Contents

  • MPEG-7
  • MPEG-21
  • Metadata Generation & Annotation
  • Social Software & Metadata
slide-47
SLIDE 47

http://www.uni-klu.ac.at

47

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Social Software

  • Social Software

 Integration of the User  Common in the Web 2.0  User participate

  • Social aspects

 U. connect to users -> Social Networking  U. connect to information -> 43things.com  U. connect to resources -> social bookmarking  U. connect to media -> social media sharing

slide-48
SLIDE 48

http://www.uni-klu.ac.at

48

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Example: Social Bookmarking

Social Bookmarking defined:

  • Bookmarking Resources
  • Providing a „stream of bookmarks“
  • Eventually additional support for

 Tagging (keywords)  Caching (Saving the state of the bookmark)  Organization & Collaboration (Groups)

slide-49
SLIDE 49

http://www.uni-klu.ac.at

49

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Example: del.icio.us

slide-50
SLIDE 50

http://www.uni-klu.ac.at

50

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Example: del.icio.us

Popularity Timeliness Syndication Navigation

slide-51
SLIDE 51

http://www.uni-klu.ac.at

51

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Example: del.icio.us

  • User Interface

 Clean and easy2use  Powerful tools (bookmarklets & plugins)

  • Additional Features

 Thumbnails  Social Networking

slide-52
SLIDE 52

http://www.uni-klu.ac.at

52

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

del.icio.us

  • User intentions are unclear:

 Self-organization or group organization  Participation / Being part of it

  • Explicitly Generated

 Bookmarking & Tagging  Tag Bundles

  • Implicitly Generated

 Time, Interestingness, The „Seen Web“  User Profile, Social Network

slide-53
SLIDE 53

http://www.uni-klu.ac.at

53

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Examples: Social Media Sharing

  • Flickr.com, Bubbleshare.com,

Zooomr.com, ...

 Sharing images & annotations

  • YouTube.com, Google Video,

VideoEgg.com. ...

 Sharing videos & annotations

  • Pandora, Last.fm

 Sharing music & flavors

slide-54
SLIDE 54

http://www.uni-klu.ac.at

54

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Example: Google Video

slide-55
SLIDE 55

http://www.uni-klu.ac.at

55

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Metadata in Social Software?

  • Bottom up

 In contrast to controlled vocabularies  In contrast to quality ensured content creation processes

  • Superimposed structure

 Instead of using predefined hierarchies  Through heavy use of linking / interrelation

  • Huge and fuzzy

 Unimaginable mass of links & tags  Lots of redundant information

  • Spammed

 Just starting ...

slide-56
SLIDE 56

http://www.uni-klu.ac.at

56

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Folksonomies

  • Definition & Description
  • Why do tagging systems work?
slide-57
SLIDE 57

http://www.uni-klu.ac.at

57

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Folksonomies

Network of Tags, Users and URLs

  • Users describe resources
  • By using (multiple) tags

Examples:

  • Social bookmarking, media sharing, etc.
slide-58
SLIDE 58

http://www.uni-klu.ac.at

58

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Folksonomies: The Structure

User tags resource (URL)

  • 1+ words or phrases (bonn, „mathias lux“)
  • No controlled vocabulary, taxonomy
  • No quality control
  • No constraints (language, length, number)
slide-59
SLIDE 59

http://www.uni-klu.ac.at

59

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Folksonomies: Structure

  • Tag to URL is a n:m relation
  • Superimposed structure through

bidirectional links

  • Structure is called „folksonomy“
slide-60
SLIDE 60

http://www.uni-klu.ac.at

60

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Folksonomy Example: Flickr

slide-61
SLIDE 61

http://www.uni-klu.ac.at

61

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Folksonomy Example: Technorati

slide-62
SLIDE 62

http://www.uni-klu.ac.at

62

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Folksonomy Example: 43things

slide-63
SLIDE 63

http://www.uni-klu.ac.at

63

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Types of Folksonomies

  • Narrow folksonomies

 Each one tags her/his own resources  All above examples are narrow f.

  • Broad folksonomies

 Each tags whatever s/he wants  Example: Social bookmarking

  • Difference

 Narrow folksonomies are more sparse

slide-64
SLIDE 64

http://www.uni-klu.ac.at

64

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Why do tagging systems work?

This was topic of a panel at CHI 2006, following conclusions were drawn:

  • Tagging has a benefit for the user

 Similar to bookmarking, integrated apps  Benefit of accessibility from everywhere in the internet

  • Tagging allows social interaction

 Connecting a user to a community trough tags  People can subscribe your stream

slide-65
SLIDE 65

http://www.uni-klu.ac.at

65

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Why do tagging systems work? (2)

  • Tags are useful for retrieval

 Synonyms and typos vanish in the mass of tags  Communities can retrieve “their” stuff (e.g. by special tag)

  • Tagging Systems have a low participation

barrier

 Apps are easy to use, intuitive, responsive  Free text is used to do the tagging  Requires no previous considerations & training

slide-66
SLIDE 66

http://www.uni-klu.ac.at

66

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Folksonomy Analysis

  • Some scientific background ...

image from http://www.squaredot.com/geek.html

slide-67
SLIDE 67

http://www.uni-klu.ac.at

67

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Unified Model for Social Networks & Semantics

Mika P. (2004) “Ontologies are us: A unified model of social networks and semantics”

  • Ontologies contain instances I and

concepts C

  • Ontologies are formal specifications

 Which are stripped from their original social context of creation  Which are static and may get outdated

slide-68
SLIDE 68

http://www.uni-klu.ac.at

68

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Where do semantics emerge from?

A third set besides C and I is needed

  • Agents A are those who specify
  • Agent defines

 which Concept C is  assigned to Instance I

⇒ A tripartite model can be identified

slide-69
SLIDE 69

http://www.uni-klu.ac.at

69

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

A tripartite model

  • 3 partitions: A, C & I
  • Hyperedges connect exactly one a ∈ A

with one c ∈ C and i ∈ I

  • One edge denotes that a user assigns a

concept to a resource.

A C I

But tripartite graphs are rather hard to understand and to work with!

slide-70
SLIDE 70

http://www.uni-klu.ac.at

70

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Simplifying the tripartite Model

Similar to the introduced structure of folksonomies:

  • An instance is connected to a concept

 like a tag to a resource

  • The edge is labeled by the user or
  • Weighted by the number of assignments
slide-71
SLIDE 71

http://www.uni-klu.ac.at

71

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

A bipartite Model ...

A graph connecting

  • Instances i to
  • Concepts c

We call this IC-Graph The weights can be expressed in an association matrix

c1 c2 c3 ... i1 1 5 ... i2 3 ... i3 4 2 2 ... ... ... ... ... ...

slide-72
SLIDE 72

http://www.uni-klu.ac.at

72

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

The Association Matrix

  • This matrix connects two different sets
  • Folding allows to transform the Matrix to a
  • ne mode network
  • Just like the co-occurence matrix in text

retrieval:

  • Result is a matrix connecting concepts to

concepts

c IC IC

M M M

slide-73
SLIDE 73

http://www.uni-klu.ac.at

73

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Example: Concepts

computer pda cellphone wlan network i1 7 5 6 1 i2 7 1 1 1 2 i3 4 5 i4 8 6 i5 3 3 4

computer pda cellphone wlan network computer 111 62 20 62 60 pda 62 56 9 68 28 cellphone 20 9 41 12 wlan 62 68 100 24 network 60 28 12 24 34

slide-74
SLIDE 74

http://www.uni-klu.ac.at

74

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

The Association Matrix

  • Also instance based co-occurrence can be

calculated

  • Based on the co-occurrence clustering

algorithms can be applied:

 Instance Clustering  Concept Clustering

I IC IC

M M M

slide-75
SLIDE 75

http://www.uni-klu.ac.at

75

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Other Association Matrices

  • Based on the AC-Graph

 Bipartite agent2concept graph  Instances are used as weights

  • Based on the AI-Graph

 Bipartite agent2instance Graph  concepts are used as weights

  • Based on A[C|I]-Graph the social network

between agents can be analyzed

slide-76
SLIDE 76

http://www.uni-klu.ac.at

76

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Application to Folksonomies

  • Concepts, agents and instances in

Folksonomies:

 Tags are concepts  Agents are users  Resources are instances

  • Tags are error prone, but semantics can

eventually emerge (see P. Mika for the example del.icio.us)

slide-77
SLIDE 77

http://www.uni-klu.ac.at

77

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Problems of the approach

  • Community based concepts & associations
  • Tags have typos, synonyms
  • Tags have different intentions

 Abstract semantics (funny, sad, friendship)  Media description (pdf, online, word, image)  Rights and authors (persons names)  Organizational (2read, todo, marker)  etc.

slide-78
SLIDE 78

http://www.uni-klu.ac.at

78

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Problems of the approach

  • Computational problems

 Big matrix multiplications are hard to compute

  • Narrow folksonomies restrict tagging to

the originating user:

 Flickr tags can only be assigned by the uploader  YouTube has the same restriction

slide-79
SLIDE 79

http://www.uni-klu.ac.at

79

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Folksonomy Analysis Example

slide-80
SLIDE 80

http://www.uni-klu.ac.at

80

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Tag Gathering: del.icio.us

  • Based on RSS feeds of del.icio.us

 Read main feed  Get entries for each user

  • Avoid spamming

 Use entries on URIs with a min. of 2 users

  • Write to relational database

 In this case MySQL 5.1

slide-81
SLIDE 81

http://www.uni-klu.ac.at

81

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Tag Database

entries PK usernameID PK,I1 urlID title creationdate description entry2tag PK usernameID PK,I2,I1 urlID PK,I3,I1 tagID tags PK id I1 name urls PK id I1 name usernames PK id I1 name

slide-82
SLIDE 82

http://www.uni-klu.ac.at

82

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Tag database issues

  • Group by & Having, Indexes
  • Memory (temp tables)
  • MySQL is just like Oracle:

 tune it or leave it.

  • Sample statement - Top tags:

SELECT COUNT(e.tagid), t.name, t.id FROM entry2tag e, tags t WHERE t.id = e.tagid GROUP BY e.tagid ORDER BY 1 DESC

slide-83
SLIDE 83

http://www.uni-klu.ac.at

83

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Tag similarity

  • Tags are assigned to resources
  • Tags describe same URIs-> Similarity

 E.g. Javascript & Ajax  E.g. Windows & Software  E.g. Linux & Kernel

  • Tags never describe same URIs->

Dissimilarity

 E.g. Free & Shop  E.g. Usability & SAP

slide-84
SLIDE 84

http://www.uni-klu.ac.at

84

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Tag Merging: Objectives

  • Main problems within del.icio.us (and

possibly in many folksonomies due to their nature)

 Synonyms  Basic level variation

  • Encounter these problems by “merging”

synonyms

 Different spellings: e.g. eLearning & e-Learning  Typos & plurals

slide-85
SLIDE 85

http://www.uni-klu.ac.at

85

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Tag Networks: Objectives

  • What is the conceptual structure within a

community?

  • Which tags are similar / interconnected?
  • Direction of the connection?
  • Probability of transition for network edges?
  • Network Analysis?

 Hubs, central authorities  Clusters

slide-86
SLIDE 86

http://www.uni-klu.ac.at

86

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Tag Centrality: Objectives

  • Which are the most prominent nodes?
  • Based on different measures?

 In degree  In Betweenness  PageRank / HITS

  • The removal of central nodes would hit the

connectivity hard!

slide-87
SLIDE 87

http://www.uni-klu.ac.at

87

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Tag Clustering: Objectives

  • What are interesting conceptual clusters?

 {design, webdesign, graphics}  {html, xhtml, css}  {ajax, javascript, prototype, script.aculo.us}

  • What is a meaningful disambiguation of a

topic / tag?

slide-88
SLIDE 88

http://www.uni-klu.ac.at

88

ITEC, Klagenfurt University, Austria – Multimedia Information Systems

Thanks ..

... for your attention