The Smart Personal Assistant: The Smart Personal Assistant: An - - PowerPoint PPT Presentation
The Smart Personal Assistant: The Smart Personal Assistant: An Overview An Overview Wayne Wobcke Anh Nguyen, Van Ho, Alfred Krzywicki, Anna Wong School of Computer Science and Engineering University of New South Wales Outline Outline
1/78
2/78
3/78
4/78
5/78
6/78
7/78
8/78
9/78
10/78
11/78
12/78
13/78
14/78
15/78
16/78
17/78
18/78
19/78
20/78
21/78
22/78
23/78
24/78
25/78
26/78
27/78
28/78
Graphical Interface
Speech Recognizer Text-to-Speech Engine Text Speech User Device e.g. PDA Calendar Agent Calendar Server E-Mail Agent E-Mail Server
Partial Parser
29/78
30/78
31/78
32/78
Question, declaration, imperative, … connective type subject predicate direct object indirect object complement phrase Expresses the relation of the clauses Syntactic subject Main verb Main object of the predicate Possible second object Other information e.g. time, location clause clause
33/78
34/78
trigger revise deliberation revise
35/78
36/78
TTS Engine Speech Recognizer Partial Parser Semantic Analysis INPUT Graphical Actions Text Speech OUTPUT Graphical Actions Text Speech CALENDAR AGENT E-MAIL AGENT COORDINATOR Pragmatic Analysis Response Generation E-Mail Task Processing Calendar Task Processing
37/78
38/78
39/78
Act Handling plans, Reference Resolution, Task Type Determination, People Determination, Clarification Generation, Graphical Action Handling
40/78
41/78
42/78
43/78
44/78
trigger revise deliberation revise learn
45/78
46/78
47/78
Alkemy Learner Return Response Meta-Plan Return Message List Return Messages Sorted by Sender Return Message Sub-list Return Message Summary Return Message Content Task Processing User Intention Identification Preference Processing Return Messages Sorted by Folder Return Messages Sorted by Priority
Request User Clarify
Preferences
update query
48/78
Individual = Device x Task x Mode x (Set Email) x PlanName
Class: true/false
Individual -> Class
Email = Sender x Length x Folder x Priority . . . Device: PDA, Desktop; Mode: Speech, Text; Task: Search, Read, Show, Notify;
49/78
projLength: Email -> Int; projLength: project(1);
setMsgExists: (Email -> Bool) -> (Set Email) -> Bool; setMsgExists: setexists(1);
and (projMsgs o numOfMsgs(true) o eq1) (projMsgs o setMsgExists(projLength o lt30));
50/78
top >-> projDevice o top; top >-> projMode o top; top >-> and (projMsgs o numOfMsgs(true) o eq0) (top); top >-> and (projMsgs o numOfMsgs(true) o eq1) (projMsgs o setMsgExists(projPriority o top)); top >-> eqDevicePDA; top >-> eqModeSpeech; top >-> eqPriorityHIGH; top >-> lt30;
51/78
User Is there any new mail from Wayne? SPA You have one new message from Wayne Wobcke. The message is more than thirty lines, should I just show you the summary? User Yes please. SPA <Displays summary of the message from Wayne Wobcke> SPA learns to display only the summary if the message length is more than thirty lines.
52/78
User Find all messages about meeting in the Inbox. SPA There are twenty messages about meeting in your Inbox. I'm displaying the first ten messages. <Displays the first ten message headers> SPA has learned to show only the first ten message headers if there are fifteen or more messages in the result.
53/78
User Show me the one from John. SPA Here is the summary of the message from John Lloyd. <Displays summary of the message from John Lloyd> SPA has learned the user’s preferences: display the message summary if the message is not of high priority and its length is more than thirty lines.
54/78
numOfMsgs = 0 plan = ReturnResponse numOfMsgs < 15 msgLength > 30 numOfMsgs = 1 True False plan = SortedBySender plan = ReturnSubList plan = ReturnContent plan = ReturnSummary priority = HIGH True False True False True yes n
ye s yes yes yes yes yes yes yes no no no no no no no no no
55/78
56/78
57/78
58/78
59/78
60/78
User Find e-mails about Israel. Fined e-mails about Israel. [e-mail/Israel] SPA You have 4 messages about Israel. User Delete all these e-mails. Delete all these e-mails. [ALL e-mail] SPA Are you sure you want to delete those messages? User Yes. Yes. SPA Messages have been deleted. Need better confirmations, e.g. delete those 15 messages
61/78
62/78
63/78
64/78
65/78
match names to address book, dynamically update vocabulary
66/78
67/78
suggesting attributes of structured objects
68/78
69/78
Rules crc ⇒ 401k & Wed & 10:30 & anna,anh,wayne,alfred 401k ⇒ 90min
70/78
Rules crc ⇒ 401k & Wed & 10:30 & anna,anh,wayne,alfred 401k ⇒ 90min New crc project meeting
71/78
Rules crc ⇒ 401k & Wed & 10:30 & anna,anh,wayne,alfred 401k ⇒ 90min New crc project meeting Refinement of crc rule crc & semester ⇒ 401k & Tue & 10:30 & anna,anh,wayne,alfred
72/78
Rules crc ⇒ 401k & Wed & 10:30 & anna,anh,wayne,alfred 401k ⇒ 90min New crc project meeting Refinement of crc rule crc & semester ⇒ 401k & Tue & 10:30 & anna,anh,wayne,alfred Suggest attributes for crc & semester
73/78
Rules crc ⇒ 401k & Wed & 10:30 & anna,anh,alfred,wayne 401k ⇒ 90min New crc project meeting Refinement of crc rule crc & semester ⇒ 401k & Tue & 10:30 & anna,anh,wayne,alfred Suggest attributes for crc & semester New (conflicting) rule crc & semester ⇒ 401k & Tue & 10:30 & anh,wayne,alfred
74/78
Rules crc ⇒ 401k & Wed & 10:30 & anna,anh,alfred,wayne 401k ⇒ 90min New crc project meeting Refinement of crc rule crc & semester ⇒ 401k & Tue & 10:30 & anna,anh,wayne,alfred Suggest attributes for crc & semester New (conflicting) rule crc & semester ⇒ 401k & Tue & 10:30 & anh,wayne,alfred Suggest attributes for crc & semester
75/78
Rules crc ⇒ 401k & Wed & 10:30 & anna,anh,alfred,wayne 401k ⇒ 90min New crc project meeting Refinement of crc rule crc & semester ⇒ 401k & Tue & 10:30 & anna,anh,wayne,alfred Suggest attributes for crc & semester New (conflicting) rule crc & semester ⇒ 401k & Tue & 10:30 & anh,wayne,alfred Suggest attributes for crc & semester User selects desired attributes
76/78
ranking suggestions
77/78
78/78