Mixed-Initiative Application for Equipment Diagnostics Bill - - PowerPoint PPT Presentation

Bill Cheetham Bill Cheetham General Electric Global Research General Electric Global Research

Outline:

• Appliances Help Desk Application
• Comments on Human and Software Agent Initiative
• Self-Confidence in a Diagnostic System

Mixed-Initiative Application for Equipment Diagnostics

Aircraft Engines Aircraft Engines Capital Services Capital Services Consumer Products Consumer Products Industrial Systems Industrial Systems Medical Systems Medical Systems Plastics Plastics Power Systems Power Systems Specialty Materials Specialty Materials Transportation Systems Transportation Systems NBC NBC

General Electric Overview

Niskayuna, NY – World Headquarters

Shanghai, China Bangalore, India Munich, Germany

Research Locations

Increase the ability of an Factory Service (FS) Calltaking Rep to solve a consumer product complaint on phone without scheduling service. Save The Call Challenges Limited supply of qualified people (currently 240 - 260 FS reps) High turnover of ASI reps ( ~70% yearly) Little time for training of reps Increasing demands on call reps Corporate Image depends on service

Appliances Help Desk

Case Title Case Description Questions What type of appliance is it? Can you make ice? Is the lever arm in the up position? Action Refrigerator - No Ice Refrigerator doesn’t make ice when lever arm is up Refrigerator No Yes Lower lever arm

Case-Based Reasoning (CBR) with Question Answering and Rule-Based System

Inference (eGain) k-commerce tool

Rules answer questions based on keywords or questions. Keywords:

If keyword Model_number = **7***** then question “What is the type of your refrigerator?” = Monogram

Questions:

If question “Where is the freezer on your refrigerator?” = Top_Mount then question “Does your refrigerator have an ice maker?” = False

These rules specify when to take initiative

Developer Interface - Rules

Calltaker collects information on call using Calltaking 32

Desktop launches CBR and sends brand, Product line and Problem statement to CBR.

Calltaker attempts to save the call using CBR

• Unlimited number of cases
• Only relevant cases displayed
• Attachments / Graphics

User Interface

Suggest Attribute Suggest Feature

User Interface

Outline:

• Appliances Help Desk Application
• Comments on Human and Software Agent Initiative
• Self-Confidence in a Diagnostic System

What initiative do you want from a person?

• Go beyond just doing requested tasks
• Example: Outstanding employees show initiative

Requirements for initiative

• Understanding of goals and priorities
• Understanding of current situation
• Identification of task from current situation that can help goals
• Identification of potential problems from doing task
• Confidence that task should be performed
• high – do task
• low – don’t do task
• medium – ask if task should be done
• Ability to perform task
• Inform others task has been done

Human Initiative

Software Agent Initiative

Dangers of software agent initiative

• Bothering the user

Questions about software agent initiative

• Is a person with initiative but no common sense dangerous?
• Can a software agent with initiative be dangerous?
• Do software agents have common sense?
• Locking the user out (stealing cycles)
• Acting in an unknown way (possibly undoing users actions)
• Making the user undo the computers actions
• Keeping the user from doing tasks (2001’s HAL)
• World domination - Terminator / Matrix

Bad Software Agent Initiative

Good Software Agent Initiative

Requirements for software agent initiative

• Understanding of goals and priorities
• Understanding of current situation
• Identification of task from current situation that can help goals
• Identification of potential problems from doing task
• Confidence that task should be performed
• high – do task
• low – don’t do task
• medium – ask if task should be done
• Ability to perform task
• Inform others task has been done

Attributes of grammar/spelling checking software agent

• Advice only given when rules violated (grammar, spelling)
• Unobtrusive
• User retains control
• User can ignore, accept, or teach

Good Software Agent Initiative

User Agent action initiative initiative action Start End Agent has High confidence

Human and Software Agent Initiative

Outline:

• Appliances Help Desk Application
• Comments on Human and Software Agent Initiative
• Self-Confidence in a Diagnostic System

The Problem

GE is using CBR to automate real world decision tasks that are currently being done by people. > Equipment Diagnosis (Appliances, Power Turbines) > Approving Financial Applications (Mortgage, Insurance) These automated systems need high accuracy They do not have to automate every problem

CBR Systems Problem Automate Action Assist User Human Problem Take Action

Old Process New Process

Solution

CBR system produces solution and confidence The confidence specifies if the predicted accuracy in solution is high enough to automate the action or not Automate the action when confidence is high

CBR Systems Problem Solution Confidence Is Confidence high? Automate Action yes Assist User no

Solution

CASE-BASE Problem RETRIEVE RETRIEVE Solution REVISE REVISE RETAIN RETAIN REUSE REUSE Similar Cases Solution And Confidence And Confidence

Output both a solution and the systems confidence that the solution is accurate

And Predict And Predict Accuracy Accuracy

Predict the accuracy of the solution in the REUSE phase

Confidence in People

I was gratified to be able to answer promptly, and I did. I said I didn't know.

• Mark Twain

The mean as concerns fear and confidence is courage: those that exceed in confidence are foolhardy, while those who lack confidence are cowardly.

• Aristotle from “The Doctrine of the Mean”

Related Work in CBR

Bruce McLaren and Kevin Ashley (ICCBR 2001) SIROCCO – provides advice on engineering ethics Proposed rules for when the program can not suggest advice (i.e. help it know what it knows) 1) If the best superficial matching case is a weak match 2) If enough of top N cases differ in their solution They found these rules improve system’s performance

Creating a Confidence Measure

1) Identify potential confidence indicators. 2) Use statistics about the case base to determine which indicators work best for calculating confidence. 3) Create an algorithm that takes the indicators and produces a confidence value

• Value is a term (high, low) for automation
• Value is number [0 – 1] for assisting users

Potential Confidence Indicators

Using k nearest neighbors to determine one best solution from a set

• Sum of similarities for retrieved cases with best solution
• Maximum similarity of a case with best solution
• Number of cases retrieved with best solution
• Sum of similarities for all other solutions (not best)
• Difference between sum of similarity for best and all other
• Number of cases retrieved with second best solution
• Percent of cases retrieved with best solution
• Sum of similarities for second best solution
• Average similarity of cases with best solution
• Average similarity of cases with second best solution
• Standard deviation of numeric solutions suggested
• Rules created from domain knowledge

Determine Which Indicators are Best

Run leave-one-out testing, printing out all indicators, plus the real solution, and if CBR’s best solution is correct Use C4.5 to determine best indicators

YES YES NO NO YES NO YES YES YES Is CBR’s solution correct? F0 F0 F0 F2 F0 F3 F0 F1 F0 Real solution F0 F0 F4 F0 F0 F0 F0 F1 F0 7) CBR’s suggested solution NA F3 F0 F2 NA F3 NA NA NA 6) CBR’s second solution 19.1 14.4 9.8 8.6 20.6 14.3 19.5 20.2 16.9 5) Difference (1 - 4) 2.3 4.6 4.2 2.3 4) Sum of similarity for all other 8 7 6 6 8 7 8 8 8 3) Number of cases with best 2.6 2.6 2.5 2.2 2.8 2.5 2.6 2.8 2.2 2) Max similarity of best 19.1 16.7 14.4 12.8 20.6 16.6 19.5 20.2 16.9 1) Sum of similarity for best 9 8 7 6 5 4 3 2 1 Case Number

Creating Confidence Algorithm

Confidence Equation:

If Difference (1-4) > 15.1 or 2nd_solution = F2 and Max_Similarity_for_Best > 8.9 then confidence is High else confidence is Low

C4.5 output:

Difference (1 – 4) <= 15.1 > 15.1 377 9% 2nd_Solution F0 F1 F2 F3 F4 25 16% 16 21% 6 66% 1 0% Max_Similarity <= 8.9 >8.9 7 84% 12 8%

Count Error Rate

Legend

Want a high count and low error rate

Start

Conclusion

Initiative is good when system is confident

end