LEARNING EVALUATION? Claire Tourmen Bear Seminar, 04/08/2014 - - PowerPoint PPT Presentation

LEARNING EVALUATION?

Claire Tourmen Bear Seminar, 04/08/2014 Berkeley

Why does it matter?

A « near profession »? (Worthen, 1994) In Education, systematized since the XIXe Century (Foucault, 1975) « Program Evaluation » since the 1960’s:

« Evaluation is the systematic collection of information about the activities, characteristics, and results of programs to make judgments about the program, improve or further develop program effectiveness, inform decisions about future programming, and/or increase understanding.” (Patton, 2008, p. 39).

• Public organizations and civil servants, NGO’s, private consultants,

independent experts in various fields

• + 50 societies in the world (Lavelle & Donaldson, 2009)
• American Evaluation Association, French Evaluation Society
• Accreditation process in Canada since 2010
• Journals and publications
• A majority of training programs in Schools of Education or Psychology in the

US (Lavelle & Donaldson, 2009)

French Evaluation Society www.sfe-asso.fr

Founded in 1999 Around 300 members:

• 2/3 administrations, civil servants and

NGO’s,

• 1/3 evaluators from public
• r private sectors (researchers,

consultants, independant experts)

1 conference / 2 years Regional clubs and thematic groups of interest Standards for evaluation, online resources

Joint conference with Degeval, Strasbourg, European Parliament, 2008

The Canadian experience

The CES Credential Evaluator Designation program: “The designation means that the holder has provided evidence of the education and experience required by the CES to be a competent evaluator.” A list of 5 competencies:

1) Reflective pratice competencies (i.e. applies professional evaluation standards) 2) Technical pratice competencies (i.e. frames evaluation questions) 3) Situational practice competencies (i.e. identifies the interests of all stakeholders) 4) Management practice compentencies (i.e. monitors resources) 5) Interpersonal practice competencies (i.e. use negociation skills)

129 credentialized evaluators in August 2012, 248 in March 2014!

http://www.evaluationcanada.ca/site.cgi?s=5&ss=10&_lang=EN

Publications in the field of program evaluation

Best seller

Cnn, September, 16, 2009

« 10 little-known fields with great job opportunities »: Orthoptist, Creative perfumer… and Program evaluator!

« 8. Program evaluator What you do: You'll evaluate several different programs, making suggestions about changes to make them better, or whether they should even continue. You'll switch programs every few weeks (or whenever you are done evaluating), so you'll get to work with a variety of clients, whether it's a nonprofit, corporate venture or a government initiative. What you need: A bachelor's degree is sufficient, although some evaluators have a Ph.D. from specialized training programs. Salary: $56,647”

http://www.cnn.com/2009/LIVING/worklife/09/16/ cb.littleknown.jobs.opportunities/index.html?iref=24hours

My research work

• 10 years of research in French, European and international

contexts with program evaluators + trainers in vocational training

• A focus on evaluators at work in naturalistic settings: work analysis

from psychology of work/developement incl. fields observations, clinical interviews (Piaget, 1926) and « explicitation » interviews (Vermersch, 2009)

• An increased concern for the study of evaluators’ activity (Schwandt,

2005, 2008, Fitzpatrick, Christie & Mark, 2008, Donaldson & Lipsey, 2008, Kundin 2010, Hurteau et al., 2012, Allal & Mottier-Lopez, 2009)

• How is evaluation performed in real life settings? How is it

learned? How can it be taught? Can this knowledge contribute to a theorization of evaluation (Shadish, Cook & Leviton, 1991) ?

A piagetian perspective

Jean Piaget’s theory of learning (1947, 1967), as applied to adults’ professional learning at workplace (Billett, 2001,

Vergnaud, 2001), that is “The study of how people acquire

knowledge” (Mac Carthy Gallagher & Reid, 2007) To study the development of:

• 1) “organized patterns of behavior” (“schèmes”, Piaget, 1947,

Vergnaud, 2001)

• 2) the underlying systems of “concepts” and “rules” (or

“knowledge”) that structure them in a partially conscious way

A new approach to evaluation

• Beyond taxonomies of skills (King et al. 2001, English, 2002)
• Beyond paradigms (Stufflebeam, 1980, Alkin & Christie, 2004, Guba &

Lincoln, 1989, Patton, 2008)

A developmental approach:

• Before being a method or a discourse, evaluation is a practice
• “Intelligence proceeds from action” (Piaget & Inhelder, 1966/2012, p. 33) and

“learning is an internal process of construction” (Mac Carthy Gallagher &

Reid, 2007)

• Competencies = capacities to adapt one’s action to the variations of

situations

3 research studies

• Program evaluators when designing evaluations at workplace

(Tourmen, 2009): 24 practioners (novices/experienced) in France + European Commission,

task analysis, 17 « explicitation » interviews (Vermersch, 2009), 3 field observations, 4 « clinical interviews » based on a case study (Piaget, 1926)

• Experienced program evaluators when describing their past

experience (Tourmen, Berriet-Solliec & Lépicier, 2013): 9 in depth

« explicitation » interviews with 9 experienced evaluators

• Vocational trainers when evaluating competencies during practical

examinations (Tourmen et al., 2013): 1 survey + 10 interviews with trainers, 2 field

• bservations incl. 6 interviews with 2 experienced trainers + 4 students

Trainers evaluating through a practical examination: field observations

Why is evaluation so difficult? Why should it be learnt?

An everyday activity (Stufflebeam, 1980) Complexity for program evaluators:

• High stakes
• Complex objects
• Strong temporal, technical and budgetary constraints

Complexity for professional trainers:

• Multiple stakes
• Hard to observe
• Limited time and resources

Multiples ways to evaluate and a lot of choices to be made 2 to 10 years to be learnt

• I. DIFFERENCES IN

ACTIVITY

Programs evaluators Trainers

Novices were « method-oriented » (T

• urmen,

2009)

• Goals: “to judge the effects of the program,” “to add knowledge” about

it, “to draft well-written evaluative questions,” “to build precise indicators ”. “Even if it is an evaluative question, I absolutely don’t know how

it could be answered”, “I checked whether I was in line with the definition of the indicator, I was, so I carried on,” “I followed the orders,” “I think that for the formulation of the third evaluative question I didn’t get it right.” .

= few and short reasoning of anticipation, few goals aimed at and short term goals

• Situation diagnosis: « I confess that the methods, I still have trouble to

understand which method is going to lead to what, it’s still pretty vague for me », « this is the only thing (indicator) that came to my mind » = few and short reasoning of adaptation, few situations indications taken into account to decide, some were neglected

Experienced practitioners were more « results-oriented »

• Goals: compromises between several goals and the final results they

anticipated they might obtain. They tried “to judge the effects” in a way “that

works,” so “it is read” and it “has effects” on the stakeholders. They also tried “to manage without setting the place on fire” in complex actor systems, “to help to sort things out” and “to put one’s finger on the problems.”

= more and longer anticipation reasoning, more goals aimed at and long term goals

• Situation diagnosis: played with the methods according to their diagnosis
• f the situation, more concerned with the political matters rather than the

merely technical ones, more interpretations of the situation. An evaluation

manager when building evaluative questions: “The [evaluative] questions will come to the councilors, if I begin to talk about management problems, to say that there are conflicts between some and others… my question might be refused, although it is central.”

= more and longer reasoning of adaptation, more diverse situation indications taken into account to decide (a broader diagnosis)

Objects to be evaluated

Level of formalization of a program’s objectives Number, scale, novelty, schedule and level of achievement of a program’s measures Time of the expected effects Number of target audiences etc.

Program evaluation design: situations indications and parameters diagnosed to make choices (T

• urmen, 2009) ¡

People’s strategies concerning evaluation

Origin of the evaluation demands Explicit/implicit demands Clarity, scale, compatibility of the demands Attitudes toward evaluation etc.

The means to proceed with the evaluation

Scale of the resources (time and budget) Existing information about the program and its effects Capacity/will of public actors/citizens to participate Easy/difficult access to informants etc.

CHOICES

Differences in trainers’ activities

As for experienced trainers (Tourmen et al., 2013):

• They did compromises between conflicting goals (assessing the levels with

equity, maintaining a high motivation to learn, teaching…), a lot of anticipations of what would happen (« lost of them will be stressed »)

• They were looking at different criteria and indicators while evaluating, even if

they were all looking at the same key situation parameters

• They were more or less justifying their assesments depending on the case

while debating the final judgments

= Trainers also did a wide diagnosis, aimed at different goals and reasoned to anticipate and adapt

Objects to be evaluated

Average (if known) skill level of the student Success/failure of the students’ activity How the student performs activity How the student thinks in action, what he probably knows Difficulty of the practical examination situation Average level of the class Examination agenda / teaching Etc.

Trainers while evaluating skills: situations indications and parameters diagnosed to make choices (T

• urmen et al., 2013) ¡

People’s strategies concerning evaluation

Origin and professional project of the student Student’s motivation for the subject/examination Student’s strategies to gain points Contextual/legal requirements for the examination Risk of claims/reject of evaluation results Student’s level of stress Etc.

The means to proceed with the evaluation

Time available Practical resources available Other jury’s knowledge and evaluation habits Evaluation criteria check list Etc.

CHOICES

• II. DIFFERENCES IN

KNOWLEDGE EXPLAIN DIFFERENCES IN ACTIVITY

Program evaluators Trainers

Certain evaluators reason better than

• thers…
• Longer reasoning
• More conditional reasoning (« If… Then… » statements)

using « action rules » (Vergnaud, 2001) to diagnose, predict and adapt

• Taking more goals and situation indications into account

“If I observe this and this and this, then the situation is like this; If the situation is like this, then I’d better do that; If I do that, then this will happen”.

Example of expert reasoning

An experienced evaluation manager at a Regional Council building evaluative questions:

“If the evaluative question is too broad, we won’t do it, because, we [the evaluation managers], we begin to measure the consequences in terms

• f data collection… Automatically we wonder if we have the information

available, and if not, can the evaluators have it easily or not…so we restrict the field because we say “the consultants, they can’t make it”…I know what a per diem costs, and if I know that they will have to rebuild a complete data base… I know how much it is. So if I have a €90,000 evaluation and if the consultants will have to rebuild a data base, I will just have one evaluative question, otherwise they can’t sort it

• ut.” (Tourmen, 2009)

… because they have built their own knowledge on evaluation situations!

« Spontaneous theories » (Shadish, Cook & Leviton, 1991) that describe the laws governing the phenomenon Examples:

• the way people usually act and react concerning an evaluation: “you

have the managers who want it to be quick, good and clear, you have the civil servants who want evaluation to help them in financing other projects, so sometimes evaluation is an alibi, and you have the influence of consultants who promise you the earth and finally give you undefined stuff.”

• the way different types of programs usually “happen” and “work”, the

way their objectives are decided and their outputs can produce

• utcomes: “policies are not always implemented where the stakes are the

highest“.

• the ways different methods can produce different results, their cost

and time, their bias and relevance: “If there aren’t any figures, economists

don’t think it’s worth anything. They don’t trust qualitative methods.”

« Practical knowledge »

« All evaluation practitioners are nascent evaluation theorists” (Shadish,

Cook and Leviton, 1991, p. 35.)

« Practical knowledge » (Schwandt, 2008 , Fitzpatrick, Christie & Mark, 2008,

Donaldson & Lipsey, 2008), also called « knowing in practice » (Schwandt, 2005)

This knowledge appeared to be organized around 2 core concepts, relating to 2 main conceptual fields:

• The feasibility of an evaluation: the experienced practitioners observed tried

to diagnose what would be “feasible,” “measurable,” “possible” or “impossible” and “evaluable” or “hardly evaluable” in a given situation; they also tried to achieve/ prevent it with their choices;

• The legitimacy of an evaluation process and its conclusions: the

experienced practitioners also tried to diagnose and build what would be considered as “sound,” “valid,” “reliable,” “credible” or “robust” conclusions, as

• pposed to “not serious” and “shaky” conclusions; as well as “useful,” “relevant,”

“acceptable” and “independent” conclusions that would not be “rejected.”

Objects to be evaluated

Level of formalization of a program’s objectives Number, scale, novelty, schedule and level of achievement of a program’s measures Time of the expected effects Number of target audiences…

/ compared to the way different types

• f programs usually « happen »

and « work »

Program evaluation design: core practical concepts and knowledge used in action

(T

• urmen, 2009) ¡

People’s strategies concerning evaluation

Origin of the evaluation demands Explicit/implicit demands Clarity, scale, compatibility of the demands Attitudes toward evaluation…

/ compared to the way people usually act and react in this context The means to proceed with the evaluation

Scale of the resources (time and budget) Existing information about the program and its effects Capacity/will of public actors/citizens to participate Easy/difficult access to informants…

/ compared to the way different methods can produce different results FEASIBILITY LEGITIMACY Core practical concepts

As for trainers

They also expressed spontaneous theories!

• On students strategies towards evaluation:

« He managed to give me an answer on another subject to improve his results (…) That’s typical him! »; « He managed his stress much better than I thought »

• On the students’ levels :

« He says nothing but he’s not… he’s smart », « We all do it » (on a bad practice also done by professionals), « the cow was nice » (on the complexity of the situation)

• On the means to proceed with the evaluation:

« It is a bit far from real situations… but we still see things » (on the efficiency of the examination)

• III. HOW DOES THIS

KNOWLEDGE GROW THROUGH EXPERIENCE?

Program evaluators

When does it appear in discourse?

• Not when asked directly! (Tourmen, Berriet-Solliec & Lépicier, 2013)

Evaluator 3: « (…) Question: Is there something you learned about public action during this first evaluation? Evaluator 3: It don’t understand what you mean? Question: For instance, did you tell to yourself « things never happen like they are supposed to be » (like just said) or something like that that has appeared to you? Evaluator 3: Here, no, there is nothing coming back to me, no, I could’nt tell ».

• It comes to mind and words when describing past

activities because it is embedded in action (Vermersch, 2009)

• A possible « décalage » between « active thought » and

« verbal thought » (Piaget, 1932)

A causal structure of events?

What is it about?

• Level 1: assertions about a singular case. Ex: « you could see, on the map, different

implementations between what had been done in a department (eq. to county or state) in comparison to others. » (Evaluator 2)

• Level 2: generalized assertions relevant for all, many or a type of cases.

Ex: « policies are not always implemented where the stakes are the highest » (Evaluator 2)

Forms of “generalizations” (Piaget, 1947) that link event A to event B, and action X to event A or B. A causal structure linking events to causes (other events/actions) and consequences (other events/actions)? « People act like social scientists attempting to observe regularities and explain their existence » (Turiel, 1983, p. 4).

An example

Evaluator 2:

• 44 years old,
• Associate Director,
• Studied industrial economics,
• He described the evaluation of agro-environmental measures in 3

regions (seq. to US State levels),

• He expressed a lot of Level 1 and Level 2 theories on a particular

phenomenon that struck him, its causes and its consequences.

• 1. A same policy is implemented in different ways in

different aeras

• 2. A director wants to

please his Secretary

• 3. Maybe

political pressure of the Secretary’s cabinet

• 4. The Administration wants to

« sell » its measures whatever the conditions

• 5. A disjunction between

the measures and the high stake environmental zones

• 6. Sometimes, vinegrowers

were too rich to accept a State aid

Elements of theory expressed by Evaluator 2

• 9. « political logic », « it is

something usual »

• 8. «administrative logic »
• 7. « policies are not

always implemented where the stakes are the highest »

• 10. « incompatibility

problems with the stakes, therefore, relevance problems and coherence problems »

• 11. « incoherence of public actors in the implementation of a policy »

How does it grow?

• A first core idea is developed on a first case (surprise, emotions), then tested,

enriched, modified through further experience (« simple idea », Campbell & McGrath, 2011, or « p-prim », Di Sessa, 2000)

• Piaget : “It is during the construction of the rules that learning takes

place” (Mac Carthy Gallagher & Reid, 2007) and “experience is central to learning because it enables the child to modify the original theory with which he or she has begun the task” (Mac Carthy Gallagher & Reid, 2007, p. 173)

• Following probabilistic (bayesian) rules of inference? Gopnik & Wellman,

2012: « Intuitive theories are representations of the causal structure of the world (…) in way that goes beyond simple association »

• Role of others and readings in this learning process

Discussion: strengths and limits of this kind of knowledge

Process involved Strengths Limits Level 1 Level 2 Possibility to:

• Draw lessons from

experience (« learning by doing »)

• Accumulate and share

pratical knowledge Risks of:

• Generalize from

singular cases

• Ignore parts of reality

that are not directly experienced Level 2 Level 1 Ability to:

• Anticipate the state of

a new program to be evaluated (no time lost, focus on relevant indicators, quicker data interpretation…) Risk of:

• Confirmation bias

(Patton, 2008): tendancy to favor information that confirms our previous beliefs

Synthesis

Evaluators develop practical knowledge - « concepts » and « rules » (Piaget, 1947) - through experience… …that help them to reason (predict, diagnose and adapt) in a new situation. = how can we prepare, encourage and secure this learning process among young evaluators?

• IV. CONSEQUENCES

FOR THE TEACHING OF EVALUATION

Program evaluators

Obstacles to evaluation learning

List of factors (Billett, 2001) relevant for evaluation:

• Long time projects + complex and partly invisible effects

= hard to draw lessons from practice

• Variety (confusion?) of methods and guidelines

= hard to follow one simple model

“You can be manipulated by different kinds of people, make promises you cannot keep, you have to manage this and anticipate the problems. You can only do it after a lot of setbacks.”

Consequences for evaluation teaching (1)

1) Need for a first method to begin, but necessity to go beyond to learn how to read the situations and adapt to them

“If individuals learn that there is just just one way of performing tasks, this might inhibit transfer to other tasks and situations.” (Billett 2001,

• p. 86).

“The first rules are necessary for gaining initial experiences, but the rules quickly become a barrier to the learning process.” (Flyvbjerg and Sampson, 2001)

2) Engage in work activities through

guidance by others (Billett, 2001) – in class (case studies) or at workplace

Consequences for evaluation teaching (2)

3) Vary situations: “What if you had 50%

less budget? What if the program was not finished yet?” - to learn how to face « routine » but also « non routine » and more complex tasks (Billett, 2001), to develop action rules (if…then…) and use the core concepts

4) Reflect on the long term effects

• f ones’ actions: work on stories of

success and failure to link events - goals – methods – results and help to build action rules/ concepts of feasibility/legitimacy

Consequences for evaluation teaching (3)

5) Encourage the « abstraction of principles » or laws and concepts and their discussion (Billett, 2001):

“What kinds of program are the hardest to evaluate? What do people usually think about evaluation? Why do some programs fail to achieve their objectives? Do interviews and case studies provide reliable data?” Or discuss such polemic assertions as: “qualitative methods fail to provide reliable data; the objectives of a program rarely respond to social needs: nobody reads the evaluation reports” etc. Possibility to discuss practical principles with scientific concepts/knowledge on programs (political science, sociology, economy, education sciences…) to enrich peoples’ theories and fight against false/limited ones (like “agenda-setting”). Possibility to write principles on a board as a synthesis of a training

Time for discussion and questions

Thank you for your attention!

References

ALKIN, M.C. & CHRISTIE, C.A. (2004). An evaluation theory tree revisited. In ALKIN, M.C. (ed.), Evaluation Roots. Thousand Oaks: Sage. ALLAL L. & MOTTIER LOPEZ L. (2009). Au cœur du jugement professionnel en évaluation : des démarches de triangulation. Les dossiers des sciences de l’éducation, 22, 25-40. BILLETT, S. (2001). Learning in the workplace. Strategies for effective practice. Allen and Unwin: Australia. CAMPBELL, B., & MCGRATH, A.L. (2011). Where the rubber hits the road: The development of useable middle-range evaluation

• theory. In M.M. MARK, S.I. DONALDSON, & B. CAMPBELL (Eds.), Social psychology and evaluation. New York: The Guilford

Press, 346-372. DI SESSA, A. (2000). Does the mind know the difference between the physical and social worlds? In NUCCI, L., SAXE, G. & TURIEL, E. (2000). Culture, thought and development. Lawrence Erlbaum: New Jersey, 141-165. DONALDSON, S. I., & LIPSEY, M. W. (2008). Roles for theory in contemporary evaluation practice: Developing practical

• knowledge. In I. SHAW, J. GREENE, & M. MARK (Eds.), Handbook of evaluation. London: Sage.

ENGLISH, B. (2002). Competencies for evaluation practioners: where to from here ? Journal of the Australasian evaluation society, 2-2. FITZPATRICK, J. L., CHRISTIE, T. A., & MARK, M. M. (2008). Evaluation in action: Interviews with Expert Evaluators. Thousand Oaks, CA: Sage. FLYVBERG, B. & SAMPSON, B. (2001). Making social science matter: why social inquiry fails and how it can succeed again. Cambridge: Cambridge University Press. FOUCAULT, M. (1975). Surveiller et punir. Paris : Gallimard. GOPNIK, A. & WELLMAN, H.M. (2012). Reconstructing constructivism: causal models, Bayesian learning mechanisms, and the theory theory. Psychological Bulletin, 138-6, 1085-108. GUBA, E.G. & LINCOLN, Y.S. (1989). Fourth Generation Evaluation. NewburyPark: Sage Publications. HURTEAU, M, HOULE, S. & GUILLEMETTE, F. (2012). L’évaluation de programme axée sur le jugement crédible. Presses de l’Université du Québec : Québec. KING, J.A., STEVAHN, L., GHERE G., & MINNEMA, J. (2001). Toward a taxonomy of essential evaluator competencies. American Journal of evaluation, 22-2, 233-235. KUNDIN, D.M. (2010). A conceptual framework for how evaluators make everyday practice decisions. American Journal of Evaluation, 31-3, 347-362. LAVELLE, J. & DONALDSON, S.I. (2009). University-based Evaluation Training Programs in the United-States 1980-2008: an empirical examination. American Journal of Evaluation, 31(1). MAC CARTHY GALLAGHER, J. & REID, K. (2007). The learning theory of Piaget and Inhelder. iUniverse: Lincoln.

References (2)

PATTON, M.Q. (2008). Utilization focused evaluation. California : Sage, 4rth edition. PIAGET, J. (1926). La représentation du monde chez l’enfant. Paris : PUF. PIAGET, J. (1932). Le jugement moral chez l’enfant. Félix Alcan : Paris. PIAGET, J. (1947). Psychologie de l’intelligence. Paris : Colin. PIAGET, J. & Inhelder, B. (1966/2012). La psychologie de l’enfant. Paris : PUF. PIAGET, J. (1967). Logique et connaissance scientifique. Paris : Gallimard, Encyclopédie de La pléiade. SCHWANDT, T. (2005). The centrality of practice to evaluation. American Journal of Evaluation, 26-1, 95-105. SCHWANDT, T. (2008). The relevance of practical knowledge traditions to evaluation practice. In SMITH, N.L. & BRANDON, P.R. (dir). Fundamental issues in evaluation. New York: The Guilford Press, 29-40. SHADISH, W.R., COOK, T.D. & LEVITON, L.C. (1991). Foundations of Program Evaluation Theories of Practice. Newbury Park: Sage publications. STUFFLEBEAM, D. (1980). L’évaluation en éducation et la prise de decision. Ottawa: NHP. TOURMEN, C. (2009). Evaluators’ Decision Making: Between Theory, Practice and Experience. American Journal of Evaluation, 30-1, 7-30. TURIEL, E. (1983). The development of social knowledge. Morality and convention. Cambridge University Press. TOURMEN, C., DEBLAY, S., LIPP, A. & BOUDE, A. (2013). Evaluer en situation professionnelle. Rapport de recherche pour la DGER, Ministère de l’Agriculture, de l’Alimentation et de la Pêche. TOURMEN, C., BERRIET-SOLIEC, M. & LEPICIER, D. (2013). Two levels of theorization in evaluation practice. Research results. Presentation at the American Evaluation Association Conference, Washington DC, October 2013. VERGNAUD, G. (2001). Forme prédicative et forme opératoire de la connaissance. Actes du Colloque GDM 2001 « La notion de compétence en enseignement des mathématiques, analyse didactique des effets de son introduction sur les pratiques de la formation », Jean Portugais (Ed), Montréal, mai 2001. VERMERSCH, P. (2009). Describing the practice of introspection. Journal of consciousness study, 16 (10-12), 20-57. WORTHEN, B.R. (1994). Is evaluation a mature profession that warrants the preparation of evaluation professionals? New Direction for Program Evaluation, 62, Summer 1994, Jossey-Bass.