Summary Summary Object Hyperlinking j yp g Mobile Visual Search - - PDF document

MOBILE VISUAL SEARCH USING SMART M3 USING SMART-M3

Alessandro Franchi Luigi Di Stefano Tullio Salmon Cinotti Alessandro Franchi, Luigi Di Stefano, Tullio Salmon Cinotti

First International Workshop on Semantic Interoperability for Smart Spaces (SISS 2010)

June 22, 2010

Summary Summary

Object Hyperlinking

j yp g

Mobile Visual Search

Smart M3 platform

Smart-M3 platform MVS Architecture Sample Applications and Demo European Commission ARTEMIS JU SP3 SOFIA project (http://sofia-project org/) (http://sofia project.org/)

Object Hyperlinking Object Hyperlinking

Process of linking real world objects to related digital

Process of linking real-world objects to related digital

content by using some form of tag

Tags can then be read by a wireless mobile device Tags can then be read by a wireless mobile device

and information about objects and locations retrieved and displayed to the user

Object

tag

2 1

DB

3

//nj4

RFID t H dli k

www

RFID tags Hardlinks

• r SMS tags

www

Graphical tags Virtual tags (GPS coordinates)

Object Hyperlinking Object Hyperlinking

A li ti

Applications: Link an audio CD to an online website

where the songs can be previewed

SMS

g p

Enhancing a physical document (e.g.

a newspaper) with multimedia content (e g a virtual tour of a property linked

SMS tag

(e.g. a virtual tour of a property linked to a real estate advertising)

Cultural heritage and tourist

i f ti information

…

Virtual tag Graphical tags tag

Mobile Tagging Mobile Tagging

• Linking method: 2D barcodes
• Linking method: 2D barcodes

Decoding…

WWW

Take a picture Decode Website P

• Pro:

Fast recognition, even in mobile environments Cheap to produce

Th t di tl d URL ith t th d f t l d t b t

The tag can directly encode an URL without the need of a central database to

store the link

• Cons:

Requires to instrument the environment with artificial visual patterns Requires to instrument the environment with artificial visual patterns

• Examples:

Quick Response (QR), DataMatrix, Semacode, Microsoft Tag…

Mobile Visual Search Mobile Visual Search

Linking method: “natural” visual tag No artificial tagging required! …

gg g q

…

Sending…

A Scanner Darkly Plot

In a totalitarian socie ty in a near future, the undercover detective Bob Arctor is workin...

www

Display linked Send picture to Take a picture Process content processing server

Examples:

• f an object

image

Google Goggles, SnapTell, Nokia Point & Find

iCandy, Doog, kooba…

MVS principles MVS principles

C ti l i i l

Conventional principles Usability: object identification should be fast, easy

and reliable

Unobtrusiveness and productivity: deployment

should be as unobtrusive and inexpensive as possible

Additional principle Interoperability and information sharing: the Interoperability and information sharing: the

information deduced by the tag should be stored in a shared and interoperable information search extent to th b fit f thi d t li ti hi h d t the benefit of third party applications, which do not need to be aware of the connectivity nor of the identification technology used by the identification i engine

Smart-M3 Smart M3

I t bilit l tf d l d ithi SOFIA

Interoperability platform deployed within SOFIA

(Smart Objects For Intelligent Applications)

Purpose: Enabling seamless interoperability between devices

f ki d d diff f i i

• f many kinds and different manufacturer, operating in

different business domains Multi-domain Multi-device Multi-vendor

Main concepts: Main concepts: Shared tuple space mechanism for information

exchange

The interpretation of information is based on common

• ntology models

Smart-M3

Architecture

The M3 functional architecture

(Smart World) works on top of ( ) p

• ne or multiple concurrent SOA

service networks (the Service World)

Smart Spaces (SS)

World)

Smart Space

Smart Spaces (SS) Individual search extents Semantic Information Brokers (SIB)

KP

Entities for storing the information Knowledge Processors (KP) Entities capable of inserting, removing

d i i f ti f th

SIB SIB

KP

and querying information from the Smart Space

SIB

KP

MVS Architecture MVS Architecture

M3 Application:

“a scenario enabled by a set of collaborating KPs”

Modular architecture

Base module

Basic recognition scenario Uses computer vision algorithms to identify objects in

pictures

Pl i d l

Plugin modules

Address different visual search scenarios

Ontology Ontology

ItemModelType Item ImageData

Ontology classes

rdf:type rdf:type rdf:type

classes

#Item_xxx

HasItemModelType HasImageData

#ItemModelType_1 #ItemModelType_2

HasRecognizedModelType

#ImageData_yyy

HasImageURL

… “./ImageStore/Image01.png”

HasAssociatedContent

“http://...”

MVS Architecture MVS Architecture

MD MD Mobile Device Base Image Mobile Device g Station Smart Space

Plug-in KP Base KP

Space

KP

Base Image Store

Recognized Object

KP

Sample Application: M i i Maintenance scenario

…the application displays a web page with related Take a picture of the device that needs to be Object is recognized and a web page with related content such manual, most recent drivers etc… device that needs to be repaired and…

Sample Application: M bil Sh i i Mobile Shopping scenario

Take a picture Object is …and automatically added to remote p

• f an item

j recognized… added to remote shopping list Add another When finished, item , confirm order

Demo Demo Conclusions Conclusions

Novel MVS engine that uses Smart-M3 as

interoperability platform

The adoption of Smart-M3 allows for higher

interoperability between the interacting p y g entities, regardless of execution environments and implementation languages, making the and implementation languages, making the system easily extensible to previously unforeseen scenarios unforeseen scenarios

This system has been integrated in part of the

Maintenance demo developed within SOFIA Maintenance demo developed within SOFIA

Sequence Diagram

Initialization

BIS Smart Space MD Base KP MD Plugin KP User Image Store 1: join 2: subscribe

Offline initialization

3: app start 4: join 2: subscribe

initialization

5: app start j 6: join and subscribe j

Sequence Diagram

Basic Object Recognition Scenario

Image Store BIS Smart Space MD Base KP MD Plugin KP User 1: take photo 2: store

RDF

(Resource Description Framework)

I f ti i t d i th RDF

Information is represented using the RDF

format (W3C standard)

Everything described in RDF is a “resource” Each resource is described by one or more

statement in the form of triples Subject- Predicate-Object

Subject: the resource Object: a value or another resource Predicate: a property that ties the subject to the

• bject.