Andes: A case study Sasikumar M C-DAC Mumbai About Andes Domain: - PowerPoint PPT Presentation

Andes: A case study Sasikumar M C-DAC Mumbai

About Andes Domain: College physics. Quantitative problems. For students of naval academy. Only help with homework – so need to change the curricula or teaching strategy for adoption. Provides a score -> useful aid to the faculty. Work in progress for over a decade; many changes in UI and strategy. Freely available, can also download. C-DAC Mumbai

... Cognitive tutoring framework – Carnegie Learning co. Objective was to find: what teachers and students will accept, what kind of hints work, which algo would scale, and how to do fair field evaluation. 356 problems covering the intended syllabus. Available as a free physics course from OLI of CMU. C-DAC Mumbai

DEMO http://quod.lib.umich.edu/j/jep/images/3336451.0 -- a video walk through C-DAC Mumbai

C-DAC Mumbai

Domain model 550 physics rules in system. Other than standard physics rules, a lot of additional knowledge needed (e.g. Common sense, math transformations, g = 9.8 m/s^2, etc). A collection of problem solving methods (PSM) for a solution: a sequence of steps. Solution graph used to represent the space of solutions. Nodes are steps in PSM. Graph because multiple ways to solve a problem, and they overlap. Need to understand user actions and offer help. C-DAC Mumbai

C-DAC Mumbai Example- Solution Graph

... For every correct method, a PSM defined, and added to graph. Even choice of different coordinate system (x-y axis position and angle) leads to a different set of steps. Student given freedom to take steps in any order (... exceptions) – Need to find where they are in the solution graph – Plan recognition problem – very hard in general. C-DAC Mumbai

... Is the user step right? – Substitute finally expected values and check for balance. – v1 + v2 = 0. What happens if v2 is zero? – For missing variables, substitue “dummy” values. – If balance, then “green”, else “red” C-DAC Mumbai

... Instructor defines the problems and works out the equations. Andes has a built-in solver; will work out all possible solutions. – Student can also use (avoids computation errors) – Can disable if needed. If there are errors – knowledge may need revision. C-DAC Mumbai

Exceptions Variables must be defined before use. Vectors must be drawn for variables. Dialogue box for each variable to be completed. Units mandatory, in standard form. C-DAC Mumbai

Pedagogy module Resemble the classroom as much as possible -> hence focus only on homework. And user interface as close to PPH as possible [pencil and paper homework]. C-DAC Mumbai

Problem selection Problems assigned by the academy. Hence no concern on problem selection. Also no decision on completion of current lesson. C-DAC Mumbai

Monitoring and FB Granularity: single step (not wait till end of solution). Feedback: immediate at each step. Flag feedback minimally. .... Elaborate on request. Hint: (if needed) 3-step hint model: ... Pointing hint, teaching hint, bottom out hint 3-types of help: ● errors which appear to be slip leads to messages right away; learner must respond to. ● other errors get a “red” flag. Help on clicking “what-is- wrong” help. Encourages self-repair! ● Next step help: if you don't know what to do next. C-DAC Mumbai

Student Model Bayesian networks for student model. Later in Andes-2, this was dropped. Some new algos used for st.mod. [Evolving assessment of student, VanLehn 88] Since no problem selection, etc SM not very critical here. C-DAC Mumbai

UI and Scaffolding Sections on screen for problem, variables, equations and plots. Tool bars and dialogue boxes for each to ensure relevant parameters are provided. Almost no parsing requirement. Undefined variables raise “red” flag and message to define. Learner can insert steps anywhere in any order (as in PPH) – Hence the plan recognition problem. C-DAC Mumbai

.... Andes expects precision in writing, etc and also in maths and graphs. Accuracy to high degree in numericals (internal calculator provided). There is an equation solver (like calculator) built-in. Instructor can disable its use, if needed. C-DAC Mumbai

UI choice b) 3x + 7 = 25 a) 3x + 7 = 25 ____ = _____ 3x = ____ ____ = _____ x = _____ ____ = _____ x = ____ c) 3x + 7 = 25 ___ = ___ [Justification ____] x = ___ [justification __] C-DAC Mumbai

“what is wrong” help A battery of error handlers to detect various errors. Only one error chosen to focus. Producing messages for multiple errors is too complex. If a step does not match, try “edit”ing the step till it becomes green in matching with the solution-graph. The “edit”s indicate possible errors. Wrong identification of type or location of errors -> students lose faith -> be careful. C-DAC Mumbai

Next Step Help Select one possible path from the solution graph consistent with the solution so far. Next untried node in that path is next step. Choose a hint from that node. – Hint should not be on a node already done; and – must be in at least one future path. C-DAC Mumbai

Scores Number of problems solved [mostly eventually every one solves a problem..] Number of greens vs number of reds. – Useful score to faculty for grading assignments. C-DAC Mumbai

Conclusion One of the successful ITS; used in practice by naval academy for many years. Lot of important aspects addressed – pedagogic issues, domain representation, etc – Except student model Lot of material (publications, etc) to study the system. A lot of empirical evaluation – Results show significant improvement in understanding – Final exam scores rise by 1 std deviation. C-DAC Mumbai

Student Model Sasikumar M IIT Bombay

Overview What is SM and why SM? Types of SM Ways to capture and update – Examples Open student model IIT Bombay

Why Student Model? Adaptive Tutoring Personalised Student Model IIT Bombay

... Student models needed for – Determining help during problem solving – Diagnosis of errors – Choice of teaching strategy – To intervene or not – To give hint or not – Choose type of hint – Etc... Note: SM depends on other modules... IIT Bombay

Purpose Short term immediate response Adapt media and resources Long term curriculum planning Enhance learner confidence Affective Provide a sense of challenge Information Build curiosity IIT Bombay

Contents of SM A SM can contain a wide variety of information, in general: – Related to knowledge of what is being taught – Other relevant background knowledge – Emotional profile (motivation, attention level, emotional state, etc) – Learner style – Cultural parameters (incl language) IIT Bombay

Issues Granularity Dependency on other modules – No point in collecting information you cannot use! – What you collect must be usable by others. Nature of representation Purpose and mode of use Change over time etc IIT Bombay

Student Model Short term vs long term model Single observation vs multiple observation Cost of building/updating vs utility – Time and complexity IIT Bombay

Classification Many classifications based on various aspects Based on type of knowledge used by the system – If-then rules – Semantic networks – Ontologies – Concept maps – And so on And type of task performed – Concept learning – Rules and constraints – Procedures SM can usually be seen as an annotation on the domain model.. IIT Bombay

Domain Types Problem solving domains – maths, physics, trouble shooting – Relatively well understood to design SM Analytic and unverifiable domains – law, ethics – Empirical verification not usually possible Design domains – architecture, music. – Most complex and ill-structured – Hardest to define SM IIT Bombay

Classification... Based on knowledge in SM – Models with course knowledge – Models with individual characteristic independent of course ● Learning style ● Personality traits ● mood Most of the focus on “course knowledge” – Dynamic update needed – Some interest in learning style for adaptive IIT Bombay instruction.

Knowledge Models Just a number 0-100? (eg. Exam score in the subject) – A vector of numbers, for each major topic? Overlay for further granularity Bug libraries Genetic models Technologies – Bayesian networks – If-then rules – Semantic nets IIT Bombay

Overlay model Subject is a set of independent pieces. A value attached to each piece. Student knowledge subset_of expert knowledge – Hence “overlay”, “cover”, etc. IIT Bombay

Overlay model Student Knowledge Expert knowledge IIT Bombay

... problems But, student knowledge not a subset of teacher knowledge – Misconceptions – Wrong procedures – Dependencies Bug libraries – Perturbation models IIT Bombay

Bug libraries Concept of bug – Enumerated? – Whatever is not right? – Concept, step, application – Full enumeration difficult except in very small domains. Perturbation approach – Perturb the right way at each node/step – The modified version – bugs IIT Bombay