Andes: A case study Sasikumar M C-DAC Mumbai About Andes Domain: - - PowerPoint PPT Presentation

C-DAC Mumbai

Andes: A case study

Sasikumar M

C-DAC Mumbai

About Andes

Domain: College physics. Quantitative problems. For students of naval academy. Only help with homework – so need to change the curricula or teaching strategy for adoption. Provides a score -> useful aid to the faculty. Work in progress for over a decade; many changes in UI and strategy. Freely available, can also download.

C-DAC Mumbai

...

Cognitive tutoring framework – Carnegie Learning co. Objective was to find: what teachers and students will accept, what kind of hints work, which algo would scale, and how to do fair field evaluation. 356 problems covering the intended syllabus. Available as a free physics course from OLI of CMU.

C-DAC Mumbai

DEMO

http://quod.lib.umich.edu/j/jep/images/3336451.0

• - a video walk through

C-DAC Mumbai

C-DAC Mumbai

Domain model

550 physics rules in system. Other than standard physics rules, a lot of additional knowledge needed (e.g. Common sense, math transformations, g = 9.8 m/s^2, etc). A collection of problem solving methods (PSM) for a solution: a sequence of steps. Solution graph used to represent the space of solutions. Nodes are steps in PSM. Graph because multiple ways to solve a problem, and they

• verlap.

Need to understand user actions and offer help.

C-DAC Mumbai

Example- Solution Graph

C-DAC Mumbai

...

For every correct method, a PSM defined, and added to graph. Even choice of different coordinate system (x-y axis position and angle) leads to a different set

• f steps.

Student given freedom to take steps in any order (... exceptions)

– Need to find where they are in the solution graph – Plan recognition problem – very hard in general.

C-DAC Mumbai

...

Is the user step right?

– Substitute finally expected values and check for balance. – v1 + v2 = 0. What happens if v2 is zero? – For missing variables, substitue “dummy” values. – If balance, then “green”, else “red”

C-DAC Mumbai

...

Instructor defines the problems and works out the equations. Andes has a built-in solver; will work out all possible solutions.

– Student can also use (avoids computation errors) – Can disable if needed.

If there are errors – knowledge may need revision.

C-DAC Mumbai

Exceptions

Variables must be defined before use. Vectors must be drawn for variables. Dialogue box for each variable to be completed. Units mandatory, in standard form.

C-DAC Mumbai

Pedagogy module

Resemble the classroom as much as possible -> hence focus only on homework. And user interface as close to PPH as possible [pencil and paper homework].

C-DAC Mumbai

Problem selection

Problems assigned by the academy. Hence no concern on problem selection. Also no decision on completion of current lesson.

C-DAC Mumbai

Monitoring and FB

Granularity: single step (not wait till end of solution). Feedback: immediate at each step. Flag feedback minimally. .... Elaborate on request. Hint: (if needed) 3-step hint model: ... Pointing hint, teaching hint, bottom out hint 3-types of help:

• errors which appear to be slip leads to messages right

away; learner must respond to.

• other errors get a “red” flag. Help on clicking “what-is-

wrong” help. Encourages self-repair!

• Next step help: if you don't know what to do next.

C-DAC Mumbai

Student Model

Bayesian networks for student model. Later in Andes-2, this was dropped. Some new algos used for st.mod. [Evolving assessment of student, VanLehn 88] Since no problem selection, etc SM not very critical here.

C-DAC Mumbai

UI and Scaffolding

Sections on screen for problem, variables, equations and plots. Tool bars and dialogue boxes for each to ensure relevant parameters are provided. Almost no parsing requirement. Undefined variables raise “red” flag and message to define. Learner can insert steps anywhere in any order (as in PPH)

– Hence the plan recognition problem.

C-DAC Mumbai

....

Andes expects precision in writing, etc and also in maths and graphs. Accuracy to high degree in numericals (internal calculator provided). There is an equation solver (like calculator) built-in. Instructor can disable its use, if needed.

C-DAC Mumbai

UI choice

a) 3x + 7 = 25 3x = ____ x = _____ c) 3x + 7 = 25 ___ = ___ [Justification ____] x = ___ [justification __] b) 3x + 7 = 25 ____ = _____ ____ = _____ ____ = _____ x = ____

C-DAC Mumbai

“what is wrong” help

A battery of error handlers to detect various errors. Only one error chosen to focus. Producing messages for multiple errors is too complex. If a step does not match, try “edit”ing the step till it becomes green in matching with the solution-graph. The “edit”s indicate possible errors. Wrong identification of type or location of errors -> students lose faith -> be careful.

C-DAC Mumbai

Next Step Help

Select one possible path from the solution graph consistent with the solution so far. Next untried node in that path is next step. Choose a hint from that node.

– Hint should not be on a node already done; and – must be in at least one future path.

C-DAC Mumbai

Scores

Number of problems solved [mostly eventually every one solves a problem..] Number of greens vs number of reds.

– Useful score to faculty for grading assignments.

C-DAC Mumbai

Conclusion

One of the successful ITS; used in practice by naval academy for many years. Lot of important aspects addressed – pedagogic issues, domain representation, etc

– Except student model

Lot of material (publications, etc) to study the system. A lot of empirical evaluation

– Results show significant improvement in understanding – Final exam scores rise by 1 std deviation.

IIT Bombay

Student Model

Sasikumar M

IIT Bombay

Overview

What is SM and why SM? Types of SM Ways to capture and update

– Examples

Open student model

IIT Bombay

Why Student Model?

Personalised Adaptive Tutoring Student Model

IIT Bombay

...

Student models needed for

– Determining help during problem solving – Diagnosis of errors – Choice of teaching strategy – To intervene or not – To give hint or not – Choose type of hint – Etc...

Note: SM depends on other modules...

IIT Bombay

Purpose

Short term immediate response Adapt media and resources Long term curriculum planning Enhance learner confidence Provide a sense of challenge Build curiosity

Affective Information

IIT Bombay

Contents of SM

A SM can contain a wide variety of information, in general:

– Related to knowledge of what is being

taught

– Other relevant background knowledge – Emotional profile (motivation, attention

level, emotional state, etc)

– Learner style – Cultural parameters (incl language)

IIT Bombay

Issues

Granularity Dependency on other modules

– No point in collecting information you

cannot use!

– What you collect must be usable by others.

Nature of representation Purpose and mode of use Change over time etc

IIT Bombay

Student Model

Short term vs long term model Single observation vs multiple observation Cost of building/updating vs utility

– Time and complexity

IIT Bombay

Classification

Many classifications based on various aspects Based on type of knowledge used by the system

– If-then rules – Semantic networks – Ontologies – Concept maps – And so on

And type of task performed

– Concept learning – Rules and constraints – Procedures

SM can usually be seen as an annotation on the domain model..

IIT Bombay

Domain Types

Problem solving domains – maths, physics, trouble shooting

– Relatively well understood to design SM

Analytic and unverifiable domains – law, ethics

– Empirical verification not usually possible

Design domains – architecture, music.

– Most complex and ill-structured – Hardest to define SM

IIT Bombay

Classification...

Based on knowledge in SM

– Models with course knowledge – Models with individual characteristic

independent of course

• Learning style
• Personality traits
• mood

Most of the focus on “course knowledge”

– Dynamic update needed – Some interest in learning style for adaptive

instruction.

IIT Bombay

Knowledge Models

Just a number 0-100? (eg. Exam score in the subject)

– A vector of numbers, for each major topic?

Overlay for further granularity Bug libraries Genetic models Technologies

– Bayesian networks – If-then rules – Semantic nets

IIT Bombay

Overlay model

Subject is a set of independent pieces. A value attached to each piece. Student knowledge subset_of expert knowledge

– Hence “overlay”, “cover”, etc.

IIT Bombay

Overlay model

Expert knowledge Student Knowledge

IIT Bombay

... problems

But, student knowledge not a subset of teacher knowledge

– Misconceptions – Wrong procedures – Dependencies

Bug libraries

– Perturbation models

IIT Bombay

Bug libraries

Concept of bug

– Enumerated? – Whatever is not right? – Concept, step, application – Full enumeration difficult except in very

small domains.

Perturbation approach

– Perturb the right way at each node/step – The modified version – bugs