ATOM ate It! End-user Context-Sensitive Automation using - - PowerPoint PPT Presentation

▶
atom ate it
SMART_READER_LITE
LIVE PREVIEW

ATOM ate It! End-user Context-Sensitive Automation using - - PowerPoint PPT Presentation

Aug 16, 2022 519 likes •1.44k views

ATOM ate It! End-user Context-Sensitive Automation using Heterogeneous Information Sources on the Web Max Van Kleek, Brennan Moore, David Karger Max Van Kleek, Paul Andr, mc schraefel MIT CSAIL enAKTing ECS, University of Southampton {

slide-1
SLIDE 1

ATOMate It!

End-user Context-Sensitive Automation using Heterogeneous Information Sources on the Web

Max Van Kleek, Brennan Moore, David Karger MIT CSAIL { emax, zamiang, karger } @ csail.mit.edu Max Van Kleek, Paul André, mc schraefel enAKTing ECS, University of Southampton { emax, pa2, mc+www } @ ecs.soton.ac.uk

Friday, May 7, 2010

slide-2
SLIDE 2

ATOMate It!

End-user Context-Sensitive Automation using Heterogeneous Information Sources on the Web How to regain some of the time that Web 2.0 took away

Max Van Kleek, Brennan Moore, David Karger MIT CSAIL { emax, zamiang, karger } @ csail.mit.edu Max Van Kleek, Paul André, mc schraefel enAKTing ECS, University of Southampton { emax, pa2, mc+www } @ ecs.soton.ac.uk

Friday, May 7, 2010

slide-3
SLIDE 3
  • 1. motivation - can we beat overload by delegating to machines?
  • 2. atomate - reactive automation driven by web feeds
  • 3. study
  • 4. related work
  • 5. next steps

Friday, May 7, 2010

slide-4
SLIDE 4
  • 1. motivation

Friday, May 7, 2010

slide-5
SLIDE 5

"...in an information-rich world, the wealth of information means a dearth of something else: a scarcity of [ ... ] the attention of its recipients. Hence a wealth of information creates a poverty of attention and a need to allocate that attention efficiently among the overabundance of information sources that might consume it" (Simon 1971, p. 40-41)

Friday, May 7, 2010

slide-6
SLIDE 6

In 2010:

Friday, May 7, 2010

slide-7
SLIDE 7

In 2010:

peer/friend/citizen-produced content

Friday, May 7, 2010

slide-8
SLIDE 8

In 2010:

peer/friend/citizen-produced content socially and algorithmically recommended

Friday, May 7, 2010

slide-9
SLIDE 9

In 2010:

peer/friend/citizen-produced content socially and algorithmically recommended

more interesting content produced every day than we could possibly consume

Friday, May 7, 2010

slide-10
SLIDE 10

In 2010:

peer/friend/citizen-produced content socially and algorithmically recommended

more interesting content produced every day than we could possibly consume

‘social maintenance’ - updating friends

Friday, May 7, 2010

slide-11
SLIDE 11

In 2010:

peer/friend/citizen-produced content socially and algorithmically recommended

more interesting content produced every day than we could possibly consume

‘social maintenance’ - updating friends keeping on top of the world

Friday, May 7, 2010

slide-12
SLIDE 12

In 2010:

peer/friend/citizen-produced content socially and algorithmically recommended

more interesting content produced every day than we could possibly consume

‘social maintenance’ - updating friends keeping on top of the world responding to others’ needs

Friday, May 7, 2010

slide-13
SLIDE 13

In 2010:

peer/friend/citizen-produced content socially and algorithmically recommended

more interesting content produced every day than we could possibly consume

‘social maintenance’ - updating friends keeping on top of the world

more responsibilities -- professional, personal, social than we could possibly perform

responding to others’ needs

Friday, May 7, 2010

slide-14
SLIDE 14
  • P. Maes,

“Agents the Reduce Work and Information Overload” Communications of the ACM

Volume 37 , Issue 7 (July 1994)

Friday, May 7, 2010

slide-15
SLIDE 15

Friday, May 7, 2010

slide-16
SLIDE 16

music listened to running sleep desktop activity

physical locations

events documents messages travels friends/enemies

Friday, May 7, 2010

slide-17
SLIDE 17

wouldn’t it be great if computers could use all this information to do stuff for us?

Friday, May 7, 2010

slide-18
SLIDE 18

for example:

  • remind me to take out the trash when I get home on

Tuesdays...

  • bug my friend who hasn’t replied to me in 2 days...
  • send me my grocery shopping list when I arrive at

the grocery store

  • remind friends about an event I am going to attend
  • text me important emails when I am traveling

...how do we get there?

Friday, May 7, 2010

slide-19
SLIDE 19

what we need

Friday, May 7, 2010

slide-20
SLIDE 20
  • 1. a way for users to express:

what they want to happen, and when, in terms of predicates relating the states and properties of people, places + things in their world. what we need

Friday, May 7, 2010

slide-21
SLIDE 21
  • 1. a way for users to express:

what they want to happen, and when, in terms of predicates relating the states and properties of people, places + things in their world. what we need actions

Friday, May 7, 2010

slide-22
SLIDE 22
  • 1. a way for users to express:

what they want to happen, and when, in terms of predicates relating the states and properties of people, places + things in their world. what we need actions conditions

Friday, May 7, 2010

slide-23
SLIDE 23
  • 1. a way for users to express:

what they want to happen, and when, in terms of predicates relating the states and properties of people, places + things in their world. what we need actions conditions predicates

Friday, May 7, 2010

slide-24
SLIDE 24
  • 1. a way for users to express:

what they want to happen, and when, in terms of predicates relating the states and properties of people, places + things in their world. what we need actions conditions predicates properties

Friday, May 7, 2010

slide-25
SLIDE 25
  • 1. a way for users to express:

what they want to happen, and when, in terms of predicates relating the states and properties of people, places + things in their world. what we need actions conditions predicates properties entities

Friday, May 7, 2010

slide-26
SLIDE 26
  • 1. a way for users to express:

what they want to happen, and when, in terms of predicates relating the states and properties of people, places + things in their world. what we need

Controlled Natural Language Interface (CNLI) for Rules

actions conditions predicates properties entities

Friday, May 7, 2010

slide-27
SLIDE 27
  • 1. a way for users to express:

what they want to happen, and when, in terms of predicates relating the states and properties of people, places + things in their world. what we need

  • 2. a way to retrieve and interpret data from our

many heterogeneous web sources as descriptions

  • f these familiar people, places and things.

Controlled Natural Language Interface (CNLI) for Rules

actions conditions predicates properties entities

Friday, May 7, 2010

slide-28
SLIDE 28
  • 1. a way for users to express:

what they want to happen, and when, in terms of predicates relating the states and properties of people, places + things in their world. what we need

  • 2. a way to retrieve and interpret data from our

many heterogeneous web sources as descriptions

  • f these familiar people, places and things.

ATOM/RSS/REST APIs, End-user mashups + RDF

Controlled Natural Language Interface (CNLI) for Rules

actions conditions predicates properties entities

Friday, May 7, 2010

slide-29
SLIDE 29

Controlled/Constrained Natural Language Interface

Abraham Bernstein and Esther Kaufmann and Christian Kaiser and Christoph Kiefer, Ginseng: A Guided Input Natural Language Search Engine for Querying Ontologies, Jena User Conference, 2008.

previous work: for the construction of RDF KBs and queries:

Friday, May 7, 2010

slide-30
SLIDE 30

Controlled/Constrained Natural Language Interface

Abraham Bernstein and Esther Kaufmann and Christian Kaiser and Christoph Kiefer, Ginseng: A Guided Input Natural Language Search Engine for Querying Ontologies, Jena User Conference, 2008.

previous work: for the construction of RDF KBs and queries: express behaviors as rules when <something happens> do <action>

Friday, May 7, 2010

slide-31
SLIDE 31

Controlled/Constrained Natural Language Interface

Abraham Bernstein and Esther Kaufmann and Christian Kaiser and Christoph Kiefer, Ginseng: A Guided Input Natural Language Search Engine for Querying Ontologies, Jena User Conference, 2008.

previous work: for the construction of RDF KBs and queries: express behaviors as rules when <something happens> do <action>

query

Friday, May 7, 2010

slide-32
SLIDE 32

Controlled/Constrained Natural Language Interface

Abraham Bernstein and Esther Kaufmann and Christian Kaiser and Christoph Kiefer, Ginseng: A Guided Input Natural Language Search Engine for Querying Ontologies, Jena User Conference, 2008.

previous work: for the construction of RDF KBs and queries: express behaviors as rules when <something happens> do <action>

query statement

Friday, May 7, 2010

slide-33
SLIDE 33

Data streams: ATOM/RSS/REST APIs, End-user mashup for alignment + RDF

Friday, May 7, 2010

slide-34
SLIDE 34

Data streams: ATOM/RSS/REST APIs, End-user mashup for alignment + RDF

Friday, May 7, 2010

slide-35
SLIDE 35

Data streams: ATOM/RSS/REST APIs, End-user mashup for alignment + RDF

ATOM FBAPI ATOM RSS0.95 RSS0.95 GDATA RSS0.95 RSS0.95 RSS0.95

Friday, May 7, 2010

slide-36
SLIDE 36

Data streams: ATOM/RSS/REST APIs, End-user mashup for alignment + RDF

people messages

emails

events

texts

www2010 ballroom www2010 302 starbucks home

  • ffice

places

david karger max van kleek brennan moore paul andré

photos web pages ???

IMs

ATOM FBAPI ATOM RSS0.95 RSS0.95 GDATA RSS0.95 RSS0.95 RSS0.95

Friday, May 7, 2010

slide-37
SLIDE 37

Data streams: ATOM/RSS/REST APIs, End-user mashup for alignment + RDF

people messages

emails

events

texts

www2010 ballroom www2010 302 starbucks home

  • ffice

places

david karger max van kleek brennan moore paul andré

photos web pages ???

IMs

ATOM FBAPI ATOM RSS0.95 RSS0.95 GDATA RSS0.95 RSS0.95 RSS0.95

from

Friday, May 7, 2010

slide-38
SLIDE 38

Data streams: ATOM/RSS/REST APIs, End-user mashup for alignment + RDF

people messages

emails

events

texts

www2010 ballroom www2010 302 starbucks home

  • ffice

places

david karger max van kleek brennan moore paul andré

photos web pages ???

IMs

ATOM FBAPI ATOM RSS0.95 RSS0.95 GDATA RSS0.95 RSS0.95 RSS0.95

currently !located !at

from

Friday, May 7, 2010

slide-39
SLIDE 39

Data streams: ATOM/RSS/REST APIs, End-user mashup for alignment + RDF

people messages

emails

events

texts

www2010 ballroom www2010 302 starbucks home

  • ffice

places

david karger max van kleek brennan moore paul andré

photos web pages ???

IMs

ATOM FBAPI ATOM RSS0.95 RSS0.95 GDATA RSS0.95 RSS0.95 RSS0.95

currently !located !at

from

RDF

Friday, May 7, 2010

slide-40
SLIDE 40

Data streams: ATOM/RSS/REST APIs, End-user mashup for alignment + RDF

people messages

emails

events

texts

www2010 ballroom www2010 302 starbucks home

  • ffice

places

david karger max van kleek brennan moore paul andré

photos web pages ???

IMs

ATOM FBAPI ATOM RSS0.95 RSS0.95 GDATA RSS0.95 RSS0.95 RSS0.95

currently !located !at

from

RDF

end-user mashup

Friday, May 7, 2010

slide-41
SLIDE 41

Data streams: ATOM/RSS/REST APIs, End-user mashup for alignment + RDF

people messages

emails

events

texts

www2010 ballroom www2010 302 starbucks home

  • ffice

places

david karger max van kleek brennan moore paul andré

photos web pages ???

IMs

ATOM FBAPI ATOM RSS0.95 RSS0.95 GDATA RSS0.95 RSS0.95 RSS0.95

RDF Activity Streams

RSS1.0

currently !located !at

from

RDF

end-user mashup

Friday, May 7, 2010

slide-42
SLIDE 42

Friday, May 7, 2010

slide-43
SLIDE 43

sharing

production ! system RDF ! world !model CLNI

Friday, May 7, 2010

slide-44
SLIDE 44

sharing

Friday, May 7, 2010

slide-45
SLIDE 45

2 atomate

a few examples architecture + world model extending the system sharing updates to other atomates

Friday, May 7, 2010

slide-46
SLIDE 46

Example 1: Simple context-sensitive reminding Remind me to take the trash out when I get home on Tuesday evenings

Friday, May 7, 2010

slide-47
SLIDE 47

Friday, May 7, 2010

slide-48
SLIDE 48

Example 2: Travel management

When I’m traveling, warn people who are e-mailing me that I might not get back to them for a while.

Friday, May 7, 2010

slide-49
SLIDE 49

How do we tell when we’re traveling?

Friday, May 7, 2010

slide-50
SLIDE 50

How do we tell when we’re traveling?

Friday, May 7, 2010

slide-51
SLIDE 51

How do we tell when we’re traveling?

Friday, May 7, 2010

slide-52
SLIDE 52

How do we tell when we’re traveling?

going !on !now?

Friday, May 7, 2010

slide-53
SLIDE 53

How do we tell when we’re traveling? currently traveling?

going !on !now?

Friday, May 7, 2010

slide-54
SLIDE 54

How do we tell when we’re traveling? currently traveling?

Google Latitude

going !on !now?

Friday, May 7, 2010

slide-55
SLIDE 55

How do we tell when we’re traveling? currently traveling?

Google Latitude

going !on !now?

Friday, May 7, 2010

slide-56
SLIDE 56

How do we tell when we’re traveling? currently traveling?

Google Latitude

current location

going !on !now?

Friday, May 7, 2010

slide-57
SLIDE 57

How do we tell when we’re traveling? currently traveling?

Google Latitude

atomate-wifi current location

going !on !now?

Friday, May 7, 2010

slide-58
SLIDE 58

How do we tell when we’re traveling? currently traveling?

Google Latitude

atomate-wifi current location

> !100 !km !from !home?

going !on !now?

Friday, May 7, 2010

slide-59
SLIDE 59

Friday, May 7, 2010

slide-60
SLIDE 60

2 atomate

a few examples architecture, rule representation extending the system sharing updates to other atomates

Friday, May 7, 2010

slide-61
SLIDE 61

atom-smasher !

server !for !connecting ! atomates !

RDF !KB

(mozStorage)

rule !engine

(core atomate XPCOM)

location !logger

(nsIWiFi)

! !feeder

web !activity ! tracker (xulrunner mozilla seamonkey, firefox, fennec)

web !sources rule !creator !+ !inspector !UI

customized !RSS and !RDF-JSON ! feeds !for !atomates !of friends, !family, !colleagues

RSS !feeds/REST !APIs actions REST !APIs() atomate !world !model !updates RDF-JSON

Friday, May 7, 2010

slide-62
SLIDE 62

inside a rule:

when (one-shot) / whenever (repeating) ANTECEDENT (conditions for execution) AND predicate( subj-pathquery, obj-pathquery-or-val ) AND predicate2(...) AND ... CONSEQUENT (what to do) action(arg-path-or-val, arg-path-or-val..)

Friday, May 7, 2010

slide-63
SLIDE 63

rules in constrained natural language

possessives for path queries

(e.g., !my !current !location’s !address)

infix english verbs for predicates eq(number,number) !=> !“is” near(Location,Location) !=> !“near” entities represented by their label

(e.g., “David Karger”, “home”, and special pronoun “me”)

“500 Fayetteville St..”

address current location me

Friday, May 7, 2010

slide-64
SLIDE 64

rules in constrained natural language

variables represented with “any/new <type>” ?x !rdf:type !:Person !=> !“any !Person” newly created Person entity: !“new !Person” bound variables with “that <type>” “any !Person’s !birthday !is !today !email !that !person !‘happy !birthday’”

Friday, May 7, 2010

slide-65
SLIDE 65

actions in constrained natural language

actions represented as fill-in-the-blank sentences with typed blanks

[“reply !to”, ! ! !{name:”email” !type:”schemas.Email”}, ! !“with”, ! !{ !name: !“message”, !type:”schemas.String”} !] reply to email with message

Friday, May 7, 2010

slide-66
SLIDE 66

2 atomate

a few examples architecture + world model extending the system sharing updates to other atomates

Friday, May 7, 2010

slide-67
SLIDE 67

extending the system

modifying/extending atomate’s schema adding new data sources (sensors or entities)

Friday, May 7, 2010

slide-68
SLIDE 68

Friday, May 7, 2010

slide-69
SLIDE 69

predicates and actions

new JV3.schemas.AtomatePredicate( { id:"occurs_date_dows", name:"occurs on a", arg0type:"schemas.SpecificDate", arg1type:"schemas.SpecificDOW", impl:"function(JV3,x,y) { return x.getDate().getDay() == y.getDate().getDay();}" } );

“occurs on a”

new JV3.schemas.AtomateAction( { id:"tweet-action", args:[“tweet”, {type:["schemas.String"], name:'message'}], impl:"function(x) { this.TwitterHelper.tweet(x); }" } );

“tweet”

extending the system

Friday, May 7, 2010

slide-70
SLIDE 70

connecting atomates

when my currently viewed page is the same as any person’s currently viewed page

Friday, May 7, 2010

slide-71
SLIDE 71

connecting atomates

when my currently viewed page is the same as any person’s currently viewed page

Friday, May 7, 2010

slide-72
SLIDE 72

connecting atomates

when my currently viewed page is the same as any person’s currently viewed page

rss pub- sub server

update feeds to different groups

  • f friends

Friday, May 7, 2010

slide-73
SLIDE 73

3 study

Can users create rules? Perceived difficulty of use Pitfalls Ideas for fixing these problems

Friday, May 7, 2010

slide-74
SLIDE 74

Rule creation study (method)

Recruited over the web Basic demographics, sign up, 2 minute tutorial video 9 Rule creation exercises (2 time, 3 easy, 3 medium, 1 difficult) Short exit survey: On average, how difficult was it to create the rules? Was there anything that was confusing/difficult? How useful would such a system be to you? What would you use this system for? What else do you wish this system could do?

Friday, May 7, 2010

slide-75
SLIDE 75

Friday, May 7, 2010

slide-76
SLIDE 76

Friday, May 7, 2010

slide-77
SLIDE 77

Rule creation study

november 2009

33 participants recruited (26 completed) (ages 25-45) 14 had some programming experience All experienced with the Web

Friday, May 7, 2010

slide-78
SLIDE 78

Rule creation study

correct - rule expressed perfectly half-correct - rule insufficiently specific; will trigger more often than intended wrong - 1 or more incorrectly expressed clause will not fire at all, or at all as intended missing - rule not completed

Friday, May 7, 2010

slide-79
SLIDE 79

Friday, May 7, 2010

slide-80
SLIDE 80

average time to complete each rule

Friday, May 7, 2010

slide-81
SLIDE 81

What caused problems?

Friday, May 7, 2010

slide-82
SLIDE 82

perceived difficulty of creating rules

Friday, May 7, 2010

slide-83
SLIDE 83

perceived usefulness

Friday, May 7, 2010

slide-84
SLIDE 84

survey

(P4) Identifying when two locations converge (i.e. mine and a friends are close). This is like social networking, but moving it towards actual life. People could grant access to their friends to view their locations, and thus know if people are close at a given time. (P7) Reminding my friends and I that we have a shared event when we're both near each other. For example, I'm often meeting with someone and both of us want to go to the same event in an hour, but we get into a coding session and we forget about the event.

what would you use atomate for?

Friday, May 7, 2010

slide-85
SLIDE 85

(P15) When I send email to someone and I want a response, I can tell atomate to send them a reminder email in 3 days if they haven't gotten back to me or something like that.

follow-ups and coordination

(P24) Emailing or responding to people when I am in transit or unavailable (no network connectivity or in an event where my phone's silenced)

Friday, May 7, 2010

slide-86
SLIDE 86

mashups and mashup-makers

Yahoo Pipes!, IBM Mashup Center, Intel Mashmaker [R. Ennals]

end-user web customization tools

Chickenfoot [M. Bolin], Coscripter [G. Leshed]

end-user reactive behavioral systems

iCAP [T. Sohn], CAMP [K. Truong], ReBA [A. Kulkharni]

constrained NL interfaces (CLNIs)

GINO/Ginseng [A. Bernstein], Sloppy [G. Little]

4 related work

Friday, May 7, 2010

slide-87
SLIDE 87

5 ongoing work

reducing errors in rule specification

rule authoring by “picking” from your lifelog

predictability/understandability

why’d it do that? what’ll it do next time?

taking into account variable feed quality (latency)

making actions end-user authorable

Friday, May 7, 2010

slide-88
SLIDE 88

6 conclusions

By treating web feeds as sensor streams for creating a simple, unified world model, we can start derive simple but useful reactive automation that helps regular users manage the deluge of information. Plenty of challenges remain! Reducing/eliminating errors in rule specification, predictability, improving efficiency of execution, etc.

Think !beyond !newsreaders !and !visual !mashups when !designing !your !Web !APIs !and !data !feeds !

Friday, May 7, 2010

slide-89
SLIDE 89

try it!

For adventurers who use Firefox (3.6+): http://atomate.me/pre-alpha

help us improve it ~

http://code.google.com/p/list-it [List-it/Atomate Framework]

acknowledgements ~

Web Science Trust, ECS, University of Southampton

contact me ~

emax@alum.mit.edu / emax@ecs.soton.ac.uk

thanks!

Friday, May 7, 2010

slide-90
SLIDE 90

Friday, May 7, 2010