Developmental Evaluation Preparing to integrate real-time feedback - - PDF document

15#04#09' 1'

Developmental Evaluation

Preparing to integrate real-time feedback and robust sense- making into your adaptive work. April'14,'2015'

Here to There

Presented'by:' Jamie'Gamble,'Imprint'Consul@ng''

15#04#09' 2'

Our'Time'Today'

' Our'Time'Today'

Today’s(Objec.ves(

1.

To'sensi@ze'leaders'of'non#profits,'chari@es'and'networks'to' the'purpose'and'major'characteris@cs'of'developmental' evalua@on'(DE).'

2.

To'assist'par@cipants'develop'a'preliminary'evalua@on'scope'

• f'work'(ESOW)'for'DE.'

3.

To'assist'par@cipants'iden@fy'their'preferences'and'possible' next'steps'for'developing'a'budget'and'securing'an'evaluator.'

4.

To'assist'par@cipants'develop'a'preliminary'“risk”'assessment' and'response'plan'for'their'DE'work.'

5.

To'provide'an'opportunity'for'par@cipants'to'review'the' appropriateness'and'readiness'of'DE'for'their'work.' '

15#04#09' 3'

The'Tasks'of'DE:'

Today’s'Focus'

• Debate'&'nego@ate'

methods'

• Simulate'methods'

and'findings'

• Finalize'design'
• Gather'data'
• Present'results'
• Produce'Report'
• Follow'up'to'ensure'use'
• ''The'Ini@a@ve'theory'
• ''User'profile'
• ''Design/methods'
• ''Budget''
• ''Evaluator''
• Developmental'niche'
• Adap@ve'leadership'
• Evalua@ve'culture'

diagnosis' scoping'

'

adap@ve' design' ' ' adap@ve' implement# a@on'

The'Ini@a@ve'

Agenda(

• Ge`ng'Grounded:'Your'Emerging'Interven@on'
• Diagnosing'DE'Fit'
• Developmental'Evalua@on:'An'Overview'
• Scoping'Ac@vity:'Conceptualize'the'Interven@on'
• Scoping'Ac@vity:'Develop'a'User'Profile'

'Lunch'

• Scoping'Ac@vity:'Design'and'Methods'
• Scoping'Ac@vity:'Evaluator'Op@ons'
• Scoping'Ac@vity:'Budget'Op@ons'
• Risk'Assessment'
• Final'Reflec@ons:'What?'So'What?'Now'What?''

15#04#09' 4'

Ice4Breaker(

• Name'
• Organiza@on'and'Role'
• What'difference'do'you'want'to'

make?'

The(Primary(Focus:(The(Ini.a.ve(

• Debate'&'nego@ate'

methods'

• Simulate'methods'and'

findings'

• Finalize'design'
• Gather'data'
• Present'results'
• Produce'Report'
• Follow'up'to'ensure'use'
• The'Ini@a@ve'theory'
• User'profile'
• Design/methods'
• Budget'op@ons'
• Evaluator'op@ons'
• Evalua@ve'culture'
• Developmental'

context''

• Adap@ve'leadership'

diagnosis' scoping'

' adap@ve' design' '

adap@ve' Implement' #a@on'

The(Ini.a.ve(

15#04#09' 5'

Ac.vity:(Your(Emerging(Ini.a.ve(

• 1. Discuss'with'your'team'
• What'is'the'challenge'you'are'trying'

to'address?'Why'is'the'status'quo' unacceptable?'

• What'are'the'characteris@cs'of'your'

emerging'strategy'to'address'it?

• How'do'you'understand'the'

connec@on'between'your'ac@vi@es'&' the'outcomes'you'seek?'

• 2. Short'presenta@on'to'plenary''

' ' Cluster'#1:' Diagnosis'

15#04#09' 6'

Cluster(1:(Diagnosis(

• Debate'&'Nego@ate'

Methods'

• Simulate'Methods'and'

Findings'

• Finalize'Design'
• Gather'Data'
• Present'Results'
• Produce'Report'
• Follow'up'to'Ensure'Use'
• The'Ini@a@ve'Theory'
• User'Profile'
• Design'Methods'
• Budget''
• Evaluator'
• Evalua@ve'Culture'
• Developmental'Niche'
• Adap@ve'Leadership'

Diagnosis' Scoping' ' Adap@ve' Design' ' Adap@ve' Implemet# a@on'

The'Ini@a@ve'

Highlights'

• The'success'of'all'evalua@on'efforts'depends'on'an'organiza@on’s'or'

group’s'evalua@ve'culture.'How''prominently'is'learning'a'part'of' your'culture?'

• There'are'several'niches'or'contexts'for'evalua@on:'developmental,'

forma@ve'and'summa@ve.''

• There'are'several'sub#niches'(5)'for'Developmental'Evalua@on'(DE).'
• Where'organiza@ons'or'groups'have'mul@ple'contexts'opera@ng'at'

the'same'@me,'Patch'Evalua@on'might'be'necessary.''

• Developmental'contexts'and'DE'require'stewardship'or'adap@ve'

leadership'in'order'to'be'useful.'

• The'“perfect”'condi@ons'for'DE'are'likely'to'be'imperfect'for'other'

types'of'evalua@on'–'the'situa@on'is'messy,'unclear,'changing,'‘out'

• f'one’s'control’'and'non#standardized.'

15#04#09' 7'

Task'1:'Assessing'Evalua@on'Culture'

• Deliberately'seeks'evidence'on'what'is'

emerging.'

• Uses'data'to'challenge'what'it'is'doing.'
• Values'candor,'challenge'and'genuine'

dialogue.'

Self#Reflec@on'&' Examina@on'

• Makes'@me'to'learn'in'a'structured'

fashion.'

• Learns'from'mistakes'and'weak'

performance.'

• Encourages'knowledge'sharing.'

Evidence#Based' Learning'

hpp://www.cgiar#ilac.org/files/publica@ons/working_papers/ ILAC_WorkingPaper_No8_Evalua@veCulture_Mayne.pdf'

Task'2:'Assessing'Context'&'Purpose'of'DE'

Developmental'

• Ini@a@ve'is'

innova@ng'and'in' development'

• Evalua@on'is'used'

to'provide' feedback'on'the' crea@on'of'the' ini@a@ve.'

Forma@ve'

• Ini@a@ve'is'

stabilizing'and'is' being'refined'

• Evalua@on'is'used'

to'help'improve' the'ini@a@ve.'

Summa@ve''

• Ini@a@ve'is'

stabilized'and' well'#established'

• Evalua@on'is'used'

to'judge'the'merit'

• r'worth'of'the'

ini@a@ve.'

Tradi&onal*contexts*

15#04#09' 8'

Task'2'Con@nued:'Some'DE'Niches'

Pre4 Forma.ve( Model( Replica.on( Ongoing( Development( Crisis( Cross4Scale( Complexity( The'ini@a@ve'is' an'emergency' response'to'a' crisis'situa@on.' The'ini@a@ve' Is'being'created' through'a'process'

• f'trial#and#error.'

The'ini@a@ve'model' 'is'being'adopted' and'adapted'for'a' new'context.' The'ini@a@ve'is' constantly'adapted'' to'fit'an'ever#' changing'context.' The'ini@a@ve'is' designed'to'change' “systems”'at'mul@ple' levels'and'sites.'

Task'2'Con@nued:'Patch'Evalua@on'

• Some'contexts'require'

mul@ple'evalua@ons'to' help'with' developmental,' forma@ve'and' summa@ve'decisions.'

• This'is'called'Patch'

Evalua@on.'

15#04#09' 9'

Task'3:'Assessing'Stewardship'&'Adap@ve' Leadership'

The'knowledge'that'innova@on'is'an'

• ngoing'process'of'experimenta@on,'

learning'and'adapta@on.' ' The'ability'to'navigate'and'improvise'on'a' shising'landscape,''en'route'to'a'goal,'and' be'open'to'the'possibility'that'the'goal' may'change'in'response'to'new'learnings' and'opportuni@es.'' ' Coopera@on'is'always'con@ngent'and' needs'to'be'con@nually'reinforced'

Comparing'Tradi@onal'and'Stewardship'Approaches'

Resources are provided based on clear plan and sustained based on fidelity to implementation Plan the work, work the plan.

Decision-making authority is clearly defined with clear channels to obtain formal permission for deviations from plan. Resources are provided with the understanding that they need to be deployed flexibly and there is a no guarantee of desired results.

Establish a general direction and strategy, and develop specifics through learning- by- doing.

Greater authority to make decisions about development of the model without excessive interference from those not involved in the day- to-day process.

Accountability based on high fidelity implementation of the plan and/or achieving pre-determined outcomes. Accountability based on developing/ model based on results, learning about what does (not) work and shifts in context. Emphasize predictability, reliability and stability. Balance traditional management concerns with curiosity, a sense of discovery and a comfort with ambiguity. TRADITIONAL MANAGEMENT & LEADERSHIP STEWARDSHIP & ADAPTIVE LEADERSHIP

Developmental Evaluation: Using Complexity Concepts to Enhance Innovation & Use. Patton. 2008.

15#04#09' 10'

Comparing'Tradi@onal'and'Stewardship'Approaches'

The leader has the answers, is ‘in control’, knows with certainty what is required to achieve the desired outcome and can command others to work together. The steward is a facilitator among co-decision makers, a listener, a guide, a learner, and a

• partner. The steward is powerful

when he/she makes others capable of achieving their goals. TRADITIONAL MANAGEMENT & LEADERSHIP STEWARDSHIP & ADAPTIVE LEADERSHIP

Developmental Evaluation: Using Complexity Concepts to Enhance Innovation & Use. Patton. 2008.

Quick'Discussion:'Assessing'Readiness'

1. How'would'you'assess'your' culture,'context'and'leadership?' Are'the'condi@ons'right'for'DE?' Which'aspects….'

• “are'in'need'of'apen@on”'
• “can'proceed'with'cau@on”'
• “are'good'to'go”'

2. What'new'ques@ons'emerge?' ' 3. Why'do'you'think'DE'might'be' appropriate'(or'not…)?'

15#04#09' 11'

' Developmental' Evalua@on:'' A'Refresher'

'

Developmental(evalua.on:(( 7(Things(You(Need(to(Know'

15#04#09' 12'

1.(Orienta.on(4(DE(is(not(a(recipe

Developmental' evalua@on' isn’t' some' par@cular' set' of' methods' or' recipe#like' steps' to' follow.' It' doesn’t'offer'a'template'of'standard'ques@ons.'' ' “It’s(a(mindset(of(inquiry(into(how(to(bring(data( to(bear(on(what’s(unfolding(so(as(to(guide(and( develop(the(unfolding.”(( ( What' that' means' and' the' @ming' of' the' inquiry' will'depend'on'the'situa@on,'context,'the'people' involved,'and'the'fundamental'principle'of'doing' what'makes'sense'for'program'development.'

Michael'Quinn' Papon.' Developmental' Evalua@on.'2010:'' pp.75#6.'

2.(Purpose(

Developmental'evalua@on' focuses'on'developmental' ques@ons:'''

• What’s'being'developed?''
• What'informa@on'do'

people'need'to'make' decisions?'

• How'is'what’s'being'

developed'and'what’s' emerging'to'be'judged?''

• Given'what’s'been'

developed'so'far'and'what' has'emerged,'what’s'next?'

'

15#04#09' 13'

3.(Evaluator(Rela.onship(

' Tradi@onal'evalua@on'posi@ons' the'evaluators'outside'the'ac@on' to'assure'independence'and'

• bjec@vity.'

' Developmental'evalua@on' posi@ons'the'evaluator'as'an' internal'team'member,' integrated'into'the'ac@on'and'

• ngoing'interpreta@on'and'

decision#making'processes.'

4.(Evaluator(Role/Func.on(

'

The'developmental'evaluator'inquires'into' developments,'tracks'developments,'facilitates' interpreta@on'of'developments'and'their' significance,'and'engages'with'innovators,'change' agents,'program'staff,'par@cipants'in'the'process,' and'funders'around'making'judgments'about' what'is'being'developed,'how'it'is'being' developed,'the'consequences'and'impacts'of' what'has'been'developed,'and'the'next'stages''

• f'development.''

Michael'Quinn'Papon.'Developmental' Evalua@on.'2010:'pp.227.'

15#04#09' 14'

Surfaces'Issues' Framing'Concepts' Tests'Quick'Itera@ons' Tracking'Developments' Evaluator(Role((con.nued)' 5.(Evalua.on(Design(

Contrary'to'the'usual'prac@ce'in' evalua@on'of'fixed'designs'that' are'implemented'as'planned,' developmental'evalua@on'designs' can'change'as'the'innova@on' unfolds'and'changes''

Michael'Quinn'Papon.'Developmental' Evalua@on.'2010,'pp.'335#6.'

' Evalua@on'design'includes:''

• Mapping'system'dynamics'&'

interdependencies,''

• 'Iden@fying'partner'assump@ons,''
• 'Informa@on'needs'of'partners,''
• 'Establishing'decision'points''
• 'Selec@ng'meaningful'indicators'
• 'Crea@ng'space'for'co#learning'

15#04#09' 15'

6.(Reports(

Dynamic'complexi@es'don’t' slow'down'or'wait'for' evaluators'to'write'their' reports,'get'them'carefully' edited,'and'then'approved'by' higher'authori@es.'' '' Any(method(can'be'used'but' will'have'to'be'adapted'to'the' necessi@es'of'speed,'real#@me' repor@ng'and'just#in#@me,'in# the#moment'decision#making'' (Papon'2010,'p.'335#6).' ' In'developmental'evalua@on' repor@ng'is'ongoing'so'as'to' affect'collec@ve'learning.'

7.(Accountability(

' In'tradi@onal'evalua@on,' accountability'is'determined' aser'the'fact,'and'is'focused'

• n'and'directed'towards''

external'stakeholders.' ' In'developmental'evalua@on,' accountability'is'real#@me,' centered'on'the'program' team’s'commitments'to'each'

• ther'and'their'shared'need'

to'have'a'posi@ve'impact'in' an'area'of'mutual'concern.''

15#04#09' 16'

Ques@ons'

'

• What'resonates'for'you'about'how'DE'might'

contribute'to'your'work?'

• What'ques@ons'–'if'any'–'emerge'for'you'about'DE?'
• What'challenges'might'you'face'if'you'were'taking'a'

developmental'approach'to'evalua@on'of'your' ini@a@ve?' '

' ' Cluster'#2:' Scoping'

15#04#09' 17'

Cluster(2:(Scoping(

• ''''''''''''Debate'&'''

'nego@ate''''''' 'methods''''''''''' 'Simulate'methods'''' 'and'findings''''''''''''''''''''''''' 'Finalize'design'

• Gather'data'
• Present'results'
• Produce'Report'
• Follow'up'to'ensure'use'
• '''''''''Theory'of'Change'
• '''''''''User'profile'
• '''''''''Design'methods'
• '''''''''Evaluator'
• Evalua@ve'culture'
• Developmental'niche'
• Adap@ve'leadership'

diagnosis' scoping(

' adap@ve' design' ' adap@ve' implement #a@on'

The(Ini.a.ve(

' Task'1:' Conceptualizing'the' Interven@on'

15#04#09' 18'

Highlights(

• 1. Evalua@on'users'and'evaluators'need'to'work'

together'to'conceptualize'the'program’s'theory:'i.e.' specifying'the'underlying'cause#and#effect' rela@onships'of'the'program,'including'what' resources'and'ac@vi@es'are'expected'to'produce'what' results.'

• 2. There'are'a'variety'of'ways'that'have'emerged'to'

describe'an'ini@a@ve’s'logic'or'theory.'

• 3. It'is'helpful'–'but'not'necessary'–'for'an'innovator'to'

have'developed'a'descrip@on'of'an'ini@a@ve’s'logic'or' theory'before'connec@ng'with'an'evaluator.'

Some'different'ways'to'conceptualize'the' theory'of'your'ini@a@ve'

' theory'of'change'*'logical'framework'*'' program'logic'*'program'model'*' interven@on'logic'*'interven@on'model'*' chain'of'objec@ves'*'outcomes'map'*'impact' pathway'analysis'*'program'theory'*'theory'

• f'change'*'theory'of'ac@on'*'working'

hypothesis'*'hypothesis'of'record'

15#04#09' 19'

Developing'a'Logic/Theory'of'An'Ini@a@ve'

Fragmented amongst stakeholders. Unevenly plausible, do-able and testable. Developing a theory of change or working hypothesis is a dynamic (and often messy) process that occurs

• ver time through board-room discussions, on the ground experimentation, and frequent checking in,

debates and adjustment based on new learnings, shifts in context and arrival of new actors. Shared amongst stakeholders. Increasingly plausible, do-able and testable. Implicit and not documented. Explicit and documented.

Comparing'DE'and'Tradi@onal'Evalua@on'

• The'central'purpose'of'developmental'evalua@on'is'to'help'

inform'the'evolu@on'of'a'group’s'program'theory.''Indeed,' the'developments'in'a'group’s'theory'of'change'–'all'the' way'from'liple'refinements'to'radical'changes'–'are'the' progress'markers'of'developmental'evalua@on.'''

• If'and'when'the'innovators'theory'of'change'and'ac@on'

begin'to'stabilize,'they'are'ready'for'conven@onal'forma@ve' and/or'summa@ve'evalua@on,'but'not'before.'''

• Evaluators'that'fail'to'make'this'switch,'run'the'risk'of'

becoming'an'“enemy”'of'innova@on”'(Westley,'Zimmerman,' Papon'2008),'requiring''innovators'to'commit'to'a' comprehensive,'precise'and'rigid'version'of'a'model,' program'or'strategy'before'they'are'ready.'

'

15#04#09' 20'

What'is'a'Theory'of'Change?'

• An'approach'to'thinking'cri@cally'about'

what'is'required'to'bring'about'a'desired' social'change'

• A'map'#'How'a'group'of'stakeholders'

expects'to'reach'a'commonly'understood' long#term'goal'

• Based'on'our'understanding'of'the'issue'

at'the'@me'–'can'evolve'as'our' understanding'of'the'issue'evolves'

Why'use'a'Theory'of'Change?'

• Big'picture'understanding'of'an'issue'and'our'

understanding'of'what'is'required'to'support' change'process'

• Helps'stakeholders'see'where'their'specific'

ini@a@ves'fit'into'the'“big'picture”'(can'help' them'align'their'work)'

• Shows'where'a'specific'ini@a@ve'needs'to'link'to'
• thers'in'order'to'achieve'the'desired'impact'
• Helps'iden@fy'indicators'for'evalua@on'

15#04#09' 21'

Theory'of'Change'example'

Group(Discussion(

How'do'you' conceptualize'your' ini@a@ve’s'logic/ theory?' ' What'outstanding' ques@ons'do'you'have' about'the'way'you' conceptualize'your' ini@a@ve?'

15#04#09' 22'

' '

Task'2:'' Develop'a'User'Profile' '

Highlights(

• 1. One'of'the'biggest'reasons'that'evalua@ons'are'not'

used'as'much'as'they'could'be'is'because'people' design'and'implement'“cookie'cuper”'designs.'

• 2. Evalua@ons'have'different'“users”'who'want'different'

ques@ons'answered'in'order'to'make'different'kinds'

• f'decisions'(aka'“users”).''
• 3. You'can'improve'the'probabili@es'that'an'evalua@ons'

“users”'will'“use”'the'evalua@ve'data'if'you'develop'a' user'profile'and'use'it'to'shape'your'evalua@on' design.'

15#04#09' 23'

Collabora@on'Involves'Different'Partners'

• Those'commiped'to'a'change;'
• Those'resistant'to'change;'
• Those'who'have'knowledge'to'contribute;'and'
• Those'who'are'impacted'by'the'change'
• Each'of'these'par@cipants'will'be'informa@on'users'

&'may'require'different'informa@on'in'order'to' ensure'that:''

• their'understanding'of'the'situa@on'is'enriched;'
• their'partners'are'contribu@ng;'
• progress'to'the'desired'change'is'being'achieved;'and'
• When'do'the'par@cipants'require'informa@on'to'make'decisions?'

Different'data'for'different'users'for'different' decisions'(uses)'

15#04#09' 24'

Mapping'Informa@on'Flows'

'How'do'we'all' understand'the' situa@on?' 'Who'needs'what' informa@on,'when?' 'What'informa@on'will' secure'everyone’s'

• ngoing'coopera@on?''

'How'will'people' contribute'to'data' collec@on?' 'How'will'we'learn' together?' 'How'will'we'know'if' we'are'successful?' Issue Stakeholder Stakeholder Stakeholder Stakeholder Stakeholder Stakeholder

USER PROFILE Example: Mentoring Organization Adapting Traditional Program for Immigrants & Refugees

User Major Evaluation Questions What decisions will they make with the answers to these questions? What are their preferences for data and methods? Language? Communication of results? Etc. When are their windows of use?

Partners: who need information to make decisions that will affect the initiative.

Program' Development' Team' What'part'of'the' program'is'working,' for'whom,'in'what' context'and'why?' What'contribu@ons' are'being'made?' What'kind'of' volunteers'mentors'do' we'need'to'recruit?' What'parts'of'the' emerging'mentoring' program'are'“core”'and' what'needs'to'be' customized'to'different' groups.' Find'case'studies'powerful.' Want'to'use'“archetypes”'for' different'immigrant'and' refugee'groups.''Want'to' weave'together'school' system'and'program'data' into'one'evalua@on'report.' Group'meets'for' quarterly'reflec@on' session.'

Secondary Users: Those that want to be kept in the loop.

United'Way' To'what'degree'does' this'program' contribute'to'school' apendance'and'course' comple@on?' Whether'to''include'the' new'mentoring'program' as'part'of'the'UW’s' Educa@on'Pillar'–'one'of' three'of'its'emerging' Collec@ve'Impact' framework.' Want'anecdotes'to'use'in' annual'campaign.' Prefer'to'have'School'Board' confirma@on'of'changes'in' student'achievement.' Success'stories' required'in'June'to' support'annual' campaign.'Decision' whether'to'include' program'in'Pillar' funding'3'years' from'now.'

15#04#09' 25'

Start(on(Your(Profile(

User Major Evaluation Questions What decisions will they make with the answers to these questions? What are their preferences for data and methods? Language? Communication of results? Etc. When are their windows

• f use?

Primary: who need it to make decisions that will affect the initiative.

INNOVATOR (You)

Secondary Users: Those that want to be kept in the loop.

' Task'3:' Evalua@on'Design'&' Methods'

15#04#09' 26'

Highlights(

1. DE'methods'refers'to'the'process'of'gathering,'analyzing'and' making'sense'of'data'in'a'way'that'(a)'answers'evalua@on' ques@ons'and'(b)'informs'innovators'decisions'about'their' emerging'ini@a@ve.' 2. DE'is'“agnos@c”'about'the'evalua@on'design'and'methods'–' all'of'them'are'on'the'table.' 3. DE'methods'must'(at'a'minimum)'reflect'the'five'core' evalua@on'standards'of'the'Canadian'Evalua@on'Society.' 4. It'is'a'primary'task'of'Developmental'Evaluators'to'develop' evalua@on'methods'and'design'in'coopera@on'with' evalua@on'users.''

An'almost'limitless'design'op@ons'&' methods'

15#04#09' 27'

DE(Requires(Customized(Design((

'

It’s(worth(emphasizing(that(no( defini.ve(list(of(developmental( evalua.on(inquiry(approaches(can(

• r(should(be(constructed.((

( Developmental'evalua@on' crea@vely'adapts'whatever' approaches'and'methods'fit'the' complexi@es'of'the'situa@on'and' are'responsive,'appropriate,'and' credible'to'social'innovators'in'

• pening'up'new'understandings'

and'guiding'further'development.'' '

Design(Considera.ons(

DE'is'learning'by'doing,'so'design'needs'to:'

• Start'evalua@on'early'(from'the'beginning'
• f'the'ini@a@ve'if'possible)'
• Incorporate'greater'frequency'of'learning'

ac@vi@es/moments'

• Be'flexible'and'adapt'as'needed''

– including'evalua@on'indicators'and'data' collec@on'methods'

15#04#09' 28'

Canadian(Evalua.on(Standards(

1. U.lity((–'does'the'evalua@on'generate'useful'data'for'the' users?' 2. Feasibility'–'is'the'evalua@on'design'efficient'and'do’able' with'the'financial,'technical'and'@ming'constraints?' 3. Proprietary'–'is'the'evalua@on'done'in'a'way'that'is' proper,'fair,'legal,'right'and'just?' 4. Accuracy'–'to'what'extent'are'the''evalua@on' representa@ons,'proposi@ons,'and'findings,'especially' those'that'support'interpreta@ons'and'judgments'about' quality,'truthful'and'dependable?' 5. Accountable'–'to'what'extent'are'evaluators'and'evaluator' users'focusing'on'documen@ng'and'improving'the' evalua@on'processes?'

hpp://www.evalua@oncanada.ca/site.cgi? s=6&ss=10&_lang=EN' '

Group(Discussion(

' What'evalua@on'prac@ces' and'approaches'do'you' already'follow'in'your'

• rganiza@on?'

' What'prac@ces'fit'with'the' DE'approach'(for'this' ini@a@ve)?'What'would' you'need'to'change'or' modify?'

15#04#09' 29'

' ' Task'4:'' Assessing'Evaluator' Op@ons'

Highlights(

• 1. DE'requires'all'the'normal'competencies'
• f'a'normal'evaluator'…'plus'a'few'more.'
• 2. There'are'several'emerging'ways'to'

arrange'for'a'DE’er'–'i.e.'internal,'external' and'internal#external'partnership'–'each' with'their'own'strengths'and'weaknesses.'

15#04#09' 30'

Evaluator(Competencies(

'

! Reflec@ve'Prac@ce'' ! Technical'Prac@ce'' ! Situa@onal'Prac@ce'' ! Management'Prac@ce'' ! Interpersonal'Prac@ce'' ' Tradi@onal'Evaluator' Competencies' Extra'Competencies'' for'DE'

! Aware'of'the'dynamic'nature'of' innova@on'and'complexity' ! High'tolerance'for'adap@ve'processes' and'comfort'with'ambiguity' ! Strong'interpersonal'skills'to'deal'with' hyperac@ve,'short#apen@on#span,' ac@on#oriented'innovators' ! Able'to'easily'draw'on'a'larger'tool'kit'

• f'evalua@ve'methods'

! Skilled'at'real'@me/flexible'and' adap@ve'evalua@on'design'and' methods' ! Facilita@on,'listening'and'partner' management'skills'

Canadian'Evalua@on'Society'

Evaluator(Arrangements(

In#House' External' Ext’t#Int’l'

Someone'internal'to' the'

• rganiza@on'or'

group'assumes' the'role'of'DE’er.' Someone'external'to' the'

• rganiza@on'or'

group'is'contracted' to'be'the'DE’er.' An'internal'and' external'DE’er'work' in'partnership'and' assume'different'DE' roles.'

15#04#09' 31'

What'evaluator'op@on'seems' to'be'the'best(fit'for'your' situa@on'right'now?'Why?'' ' If(you'have'made'a'choice,' how'will'you'address'the' limita@ons'of'this' arrangement?''

*

Assessing(Evaluator(Op.ons(

'

Task'5:'' Budge@ng'Op@ons' for'DE' '

15#04#09' 32'

Highlights(

• 1. Evalua@on'ac@vi@es'typically'range'from'5%'to'

20%'of'an'ini@a@ve’s'budget.'

• 2. DE'contracts'are'more'easily'organized'as'long'

term'consul@ng'arrangements'than'fixed'research' designs'with'rigid'deliverables.'

• 3. There'are'at'least'three'emerging'budge@ng'
• p@ons:'retainer,'step#wise,'plan'and'adjust.'
• 4. This'new'approach'to'budge@ng'and'contrac@ng'

requires'shiss'in'typical'procurement'prac@ces.'

Amount(of(Resources(

Typically,'programs'are'advised'to' allocate'10%'to'20%'percent'of'their' program'budgets'for'evalua@on' ac@vi@es'(including'both'process'and*

• utcome'evalua@ons).'This(

percentage(may(increase(or(decrease,' depending'on'the'evalua@on'design,' the'comprehensiveness'of'the' evalua@on,'and'the'type'of'evaluator.''

Source:'hpp:// www.childtrend s.org/Files/ Child_Trends' 2007_10_01_RB _Selec@ngEvalua tor.pdf' '

15#04#09' 33'

Consul.ng(Vs.(Deliverable(Oriented( Rela.onship((

Under'op@mal'circumstances,'my' work'with'clients'is'characterized' by'a'long#term'rela@onship,' regular'interac@on'and'responsive' deliverables.'DE'requires'a' transparent'process'of'iden@fying' the'available'resources'and'then' itera@vely'discussing''how'best'to' apply'them.'Its'more'like'a' consul@ng'contract'than'a'pre# specified'evalua@on'design' contract.'Trust'and' communica@on'are'essen@al.'

hpp:// beperevalua@on.org/ sites/defalt/files/ Budge@ng%20for %20Developmental %20Evalua@on.pdf' '

Budget(Op.ons(

'

• 1. Retainer(fee(contracts(–'scope'of'work'is'open#

ended,'agree'on'total'amount,'evaluator'draws'on' pool'as'required;'new'funds'added'as'required.'

• 2. Stepwise(funding(–'evalua@on'design'and'funds'

are'nego@ated'in'steps'or'phases.'

• 3. Plan(and(adjust(–'speculate'on'the'DE'design'and'

budget'based'on'a'best#guess,'but'with'a' substan@al'con@ngency'(e.g.'25#50%).'

hpp:// beperevalua@on.org/ sites/default/files/ Budge@ng%20for %20Developmental %20Evalua@on.pdf' '

15#04#09' 34'

Organiza.onal(Implica.ons(

[Budge@ng'for'DE]'requires'quite'a' shis'in'the'procurement'processes'

• f'many'organiza@ons,'which'is'

where'in'my'honest'opinion,'so' many'evalua@ons'go'wrong.' Procurement'must'be'more'flexible' and'responsive,'and'the'focus'of' procurement'is'on'capability'rather' than'a'fixed'design.'

hpp:// beperevalua@on.org /sites/default/files/ Budge@ng%20for %20Developmental %20Evalua@on.pdf' '

Quick(Discussion(

What'budget'op@on'best' fits'your'situa@on?'Why?' ' If'you'have'already' budgeted'(or'don’t'have' la@tude'to'chose'the' approach),'how'might'you' mi@gate'any'challenges'/' risks?''

15#04#09' 35'

' ' Improving'Probabili@es'

• f'DE'Success'

Improving(Probabili.es(of(DE(Success(

Aser'today,'you'should'have:'

• a'beper'sense'of'what'DE'is'
• started'on'your'DE'scope'of'work'
• a'sense'of'what'you'prefer'in'an'

evaluator'

• a'grasp'on'designing'and'

budge@ng'for'DE' Now,'what'do'you'need'to'address'in'

• rder'to'make'your'organiza@on'more'

ready'to'embrace'DE?'

15#04#09' 36'

Improving(Probabili.es(of(DE(Success(

Use'the'Risk(Assessment( Worksheet((in'your'work' tracker)'to:''

• Assess'the'risks'or'barriers'

that'may'emerge'for'your' group'in'its'efforts'to' integrate'DE'into'its'work''

• Develop'possible'strategies'

to'mi@gate'those'risks.'

' ' ' Final'Reflec@ons'

15#04#09' 37' '

• So'what'–'if'any'–'are'the'implica@ons'for'your'

emerging'interven@on(s)'and/or'using'DE?''

" DE'is'a'perfect'fit'for'our'organiza@on:'we'can' proceed'full'steam'ahead.' " DE'has'the'poten@al'to'be'a'good'fit'for'our'

• rganiza@on,'however,'we'have'several'things'we'

need'to'do'as'we'proceed.' " DE'is'not'a'fit'at'this'@me:'we'have'some'significant' work'to'do'before'we'are'ready.' '

• What'are'the'next'steps'to'move'your'ideas'forward'

regarding'the'evalua@on'of'your'ini@a@ve?' '

Moving(forward…(

Moving(forward((cont.)…(

• What'resonates'for'your'group'(i.e.'

ques@ons,'“AHAs”,'concerns)'aser'today’s' discussions?'

15#04#09' 38'

Thank'you'

www.innoweave.ca'