Developmental Evaluation Preparing to integrate real-time feedback - PDF document

15#04#09' Developmental Evaluation Preparing to integrate real-time feedback and robust sense- making into your adaptive work. April'14,'2015' Presented'by:' Jamie'Gamble,'Imprint'Consul@ng'' H ere to T here 1'

15#04#09' Our'Time'Today' ' Our'Time'Today' Today’s(Objec.ves( To'sensi@ze'leaders'of'non#profits,'chari@es'and'networks'to' 1. the'purpose'and'major'characteris@cs'of'developmental' evalua@on'(DE).' To'assist'par@cipants'develop'a'preliminary'evalua@on'scope' 2. of'work'(ESOW)'for'DE.' To'assist'par@cipants'iden@fy'their'preferences'and'possible' 3. next'steps'for'developing'a'budget'and'securing'an'evaluator.' To'assist'par@cipants'develop'a'preliminary'“risk”'assessment' 4. and'response'plan'for'their'DE'work.' To'provide'an'opportunity'for'par@cipants'to'review'the' 5. appropriateness'and'readiness'of'DE'for'their'work.' ' 2'

15#04#09' The'Tasks'of'DE:' Today’s'Focus' • ''The'Ini@a@ve'theory' • ''User'profile' • Developmental'niche' • ''Design/methods' • Adap@ve'leadership' • ''Budget'' • Evalua@ve'culture' • ''Evaluator'' diagnosis' scoping' The'Ini@a@ve' ' ' ' adap@ve' adap@ve' • Debate'&'nego@ate' • Gather'data' methods' design' implement# • Present'results' • Simulate'methods' a@on' • Produce'Report' and'findings' • Follow'up'to'ensure'use' • Finalize'design' Agenda( Ge`ng'Grounded:'Your'Emerging'Interven@on' • Diagnosing'DE'Fit' • Developmental'Evalua@on:'An'Overview' • Scoping'Ac@vity:'Conceptualize'the'Interven@on' • Scoping'Ac@vity:'Develop'a'User'Profile' • 'Lunch' Scoping'Ac@vity:'Design'and'Methods' • Scoping'Ac@vity:'Evaluator'Op@ons' • Scoping'Ac@vity:'Budget'Op@ons' • Risk'Assessment' • Final'Reflec@ons:'What?'So'What?'Now'What?'' • 3'

15#04#09' Ice4Breaker( • Name' • Organiza@on'and'Role' • What'difference'do'you'want'to' make?' The(Primary(Focus:(The(Ini.a.ve( • Evalua@ve'culture' • The'Ini@a@ve'theory' • Developmental' • User'profile' context'' • Design/methods' • Adap@ve'leadership' • Budget'op@ons' • Evaluator'op@ons' diagnosis' scoping' The(Ini.a.ve( ' ' adap@ve' adap@ve' • Debate'&'nego@ate' • Gather'data' Implement' methods' design' • Present'results' #a@on' • Simulate'methods'and' • Produce'Report' findings' • Follow'up'to'ensure'use' • Finalize'design' 4'

15#04#09' Ac.vity:(Your(Emerging(Ini.a.ve( 1. Discuss'with'your'team' What'is'the'challenge'you'are'trying' • to'address?'Why'is'the'status'quo' unacceptable?' What'are'the'characteris@cs'of'your' • emerging'strategy'to'address'it? How'do'you'understand'the' • connec@on'between'your'ac@vi@es'&' the'outcomes'you'seek?' 2. Short'presenta@on'to'plenary'' ' ' Cluster'#1:' Diagnosis' 5'

15#04#09' Cluster(1:(Diagnosis( • The'Ini@a@ve'Theory' • Evalua@ve'Culture' • User'Profile' • Developmental'Niche' • Design'Methods' • Adap@ve'Leadership' • Budget'' • Evaluator' Diagnosis' Scoping' The'Ini@a@ve' ' ' Adap@ve' • Debate'&'Nego@ate' Adap@ve' • Gather'Data' Methods' Implemet# • Present'Results' Design' • Simulate'Methods'and' • Produce'Report' a@on' Findings' • Follow'up'to'Ensure'Use' • Finalize'Design' Highlights' • The'success'of'all'evalua@on'efforts'depends'on'an'organiza@on’s'or' group’s'evalua@ve'culture.'How''prominently'is'learning'a'part'of' your'culture?' • There'are'several'niches'or'contexts'for'evalua@on:'developmental,' forma@ve'and'summa@ve.'' • There'are'several'sub#niches'(5)'for'Developmental'Evalua@on'(DE).' • Where'organiza@ons'or'groups'have'mul@ple'contexts'opera@ng'at' the'same'@me,'Patch'Evalua@on'might'be'necessary.'' • Developmental'contexts'and'DE'require'stewardship'or'adap@ve' leadership'in'order'to'be'useful.' • The'“perfect”'condi@ons'for'DE'are'likely'to'be'imperfect'for'other' types'of'evalua@on'–'the'situa@on'is'messy,'unclear,'changing,'‘out' of'one’s'control’'and'non#standardized.' 6'

15#04#09' Task'1:'Assessing'Evalua@on'Culture' • Deliberately'seeks'evidence'on'what'is' emerging.' Self#Reflec@on'&' • Uses'data'to'challenge'what'it'is'doing.' Examina@on' • Values'candor,'challenge'and'genuine' dialogue.' • Makes'@me'to'learn'in'a'structured' fashion.' Evidence#Based' • Learns'from'mistakes'and'weak' Learning' performance.' • Encourages'knowledge'sharing.' hpp://www.cgiar#ilac.org/files/publica@ons/working_papers/ ILAC_WorkingPaper_No8_Evalua@veCulture_Mayne.pdf' Task'2:'Assessing'Context'&'Purpose'of'DE' Tradi&onal*contexts* Developmental' Forma@ve' Summa@ve'' • Ini@a@ve'is' • Ini@a@ve'is' • Ini@a@ve'is' innova@ng'and'in' stabilizing'and'is' stabilized'and' development' being'refined' well'#established' • Evalua@on'is'used' • Evalua@on'is'used' • Evalua@on'is'used' to'provide' to'help'improve' to'judge'the'merit' feedback'on'the' the'ini@a@ve.' or'worth'of'the' crea@on'of'the' ini@a@ve.' ini@a@ve.' 7'

15#04#09' Task'2'Con@nued:'Some'DE'Niches' The'ini@a@ve' The'ini@a@ve'model' The'ini@a@ve'is' Is'being'created' 'is'being'adopted' constantly'adapted'' through'a'process' and'adapted'for'a' to'fit'an'ever#' of'trial#and#error.' new'context.' changing'context.' Pre4 Model( Ongoing( Forma.ve( Replica.on( Development( The'ini@a@ve'is' The'ini@a@ve'is' designed'to'change' an'emergency' “systems”'at'mul@ple' response'to'a' levels'and'sites.' crisis'situa@on.' Cross4Scale( Crisis( Complexity( Task'2'Con@nued:'Patch'Evalua@on' • Some'contexts'require' mul@ple'evalua@ons'to' help'with' developmental,' forma@ve'and' summa@ve'decisions.' • This'is'called'Patch' Evalua@on.' 8'

15#04#09' Task'3:'Assessing'Stewardship'&'Adap@ve' Leadership' The'knowledge'that'innova@on'is'an' ongoing'process'of'experimenta@on,' learning'and'adapta@on.' ' The'ability'to'navigate'and'improvise'on'a' shising'landscape,''en'route'to'a'goal,'and' be'open'to'the'possibility'that'the'goal' may'change'in'response'to'new'learnings' and'opportuni@es.'' ' Coopera@on'is'always'con@ngent'and' needs'to'be'con@nually'reinforced' Comparing'Tradi@onal'and'Stewardship'Approaches' TRADITIONAL STEWARDSHIP & MANAGEMENT & LEADERSHIP ADAPTIVE LEADERSHIP Balance traditional management concerns Emphasize predictability, reliability and with curiosity, a sense of discovery and a stability. comfort with ambiguity. Establish a general direction and strategy, and develop specifics through learning- Plan the work, work the plan. by- doing. Greater authority to make decisions about Decision-making authority is clearly defined with development of the model without excessive clear channels to obtain formal permission for interference from those not involved in the day- deviations from plan. to-day process. Accountability based on high fidelity Accountability based on developing/ model implementation of the plan and/or based on results, learning about what does achieving pre-determined outcomes. (not) work and shifts in context . Resources are provided based on clear Resources are provided with the understanding plan and sustained based on fidelity to that they need to be deployed flexibly and there is a no guarantee of desired results . implementation Developmental Evaluation: Using Complexity Concepts to Enhance Innovation & Use. Patton. 2008. 9'

15#04#09' Comparing'Tradi@onal'and'Stewardship'Approaches' TRADITIONAL STEWARDSHIP & MANAGEMENT & LEADERSHIP ADAPTIVE LEADERSHIP The steward is a facilitator The leader has the answers, is among co-decision makers, a ‘in control’, knows with certainty listener, a guide, a learner, and a what is required to achieve the partner. The steward is powerful desired outcome and can when he/she makes others command others to work capable of achieving their together. goals. Developmental Evaluation: Using Complexity Concepts to Enhance Innovation & Use. Patton. 2008. Quick'Discussion:'Assessing'Readiness' 1. How'would'you'assess'your' culture,'context'and'leadership?' Are'the'condi@ons'right'for'DE?' Which'aspects….' “are'in'need'of'apen@on”' • • “can'proceed'with'cau@on”' • “are'good'to'go”' 2. What'new'ques@ons'emerge?' ' 3. Why'do'you'think'DE'might'be' appropriate'(or'not…)?' 10'

15#04#09' ' Developmental' Evalua@on:'' A'Refresher' ' Developmental(evalua.on:(( 7(Things(You(Need(to(Know ' 11'

15#04#09' 1.(Orienta.on(4(DE(is(not(a(recipe Developmental' evalua@on' isn’t' some' par@cular' set' of' methods' or' recipe#like' steps' to' follow.' It' doesn’t'offer'a'template'of'standard'ques@ons.'' ' “It’s(a(mindset(of(inquiry(into(how(to(bring(data( to(bear(on(what’s(unfolding(so(as(to(guide(and( develop(the(unfolding.”(( Michael'Quinn' Papon.' ( Developmental' What' that' means' and' the' @ming' of' the' inquiry' Evalua@on.'2010:'' pp.75#6.' will'depend'on'the'situa@on,'context,'the'people' involved,'and'the'fundamental'principle'of'doing' what'makes'sense'for'program'development.' 2.(Purpose( Developmental'evalua@on' focuses'on'developmental' ques@ons:''' • What’s'being'developed?'' • What'informa@on'do' people'need'to'make' decisions?' • How'is'what’s'being' developed'and'what’s' emerging'to'be'judged?'' • Given'what’s'been' developed'so'far'and'what' has'emerged,'what’s'next?' ' 12'