evaluation: Prospects, challenges, and implications for practice 29 - - PowerPoint PPT Presentation

Multi-organisational collaboration in

• perations assessment and

evaluation: Prospects, challenges, and implications for practice

29th ISMOR, 2012 Andy Williams

Operational Analysis Branch HQ Supreme Allied Commander Transformation

1 The views expressed in this article are the views of the author and do not necessarily represent the views of NATO or any other organisation

2

ACT is NATO’s leading agent for change, driving, facilitating, and advocating continuous improvement

• f

Alliance capabilities to maintain and enhance the military relevance and effectiveness of the Alliance.

Allied Command Transformation

3

ACT Trident

Strategic

Thinking Capability Development Education & Training

Interoperability Coherence

Partners + Outreach Lessons Learned Lessons Identified

Abstract

Measuring progress and results of an organisation’s activities—“assessment” in the military and “evaluation” in the civilian sector—are typically arranged as principal-agent relationships between a management, leadership, funding or accountability body, and the evaluating body. Recent practice has seen the emergence of collaboration between assessment and evaluation organisations involving more complex arrangements with either multiple principals or multiple agents, or both. Furthermore, NATO has expressed an ambition for military operations assessment organisations to work more closely with civilian evaluation agencies in conflict or fragile environments. This paper proposes a framework in which military assessment organisations can interact with civilian evaluation departments. The framework considers: the rationale for collaborating on operations assessments; the necessary pre- conditions; the challenges and transaction costs involved; and the collaboration process, including structural issues of governance and administration, mechanisms of interaction, and methodology. Ideas for empirical research and further theoretical development are presented.

Keywords: Operations Assessment, Monitoring and Evaluation, Evaluation, Collaboration

4

Operations Assessment

 Operations Assessment

 “the function that enables the measurement of

progress and results of operations in a military context, and the subsequent development of conclusions and recommendations that support decision making” (COPD, p. 5-1).

 A.k.a. campaign assessment; operational

assessment, assessment

 Many similarities to “monitoring and

evaluation”

 Assessment Capability Development Plan

 Concept development  Refinement of NATO policy and directives  Experimentation  Development of new training

 Analytical support

 Capture of current problems in assessment  Lessons identified / learned  Research projects  Literature reviews, workshops, interviews

6

Context

 Problems in operations assessment  Rationale for increased collaboration  Multi-organisational collaboration framework  Conclusions and future work 7

Agenda

8

Current challenges in Ops Assessment

Data limitations Evaluating causal impacts Methodological stovepiping Drawdowns and transition

 Abundance of security-incident related data

 Allows granular measurement of change  Directly relatable to plan elements

 Abundance of other military data and reports

 Allow granular measurement  Not easily relatable to plan elements

 Abundance of governance, development,

economic, social, regional data

 Allows retrospective, strategic level assessments  Not directly relatable to plan elements

 Conclusion: militaries are planning for governance

and development impacts, but have limited data to assess progress

9

Data limitations

10

Evaluating causal impacts

 Current military plans clearly call for evaluation

• f complex causalities, which isn’t possible

without access to relevant non-military data

 Military missions eventually drawdown  Political requirements for assessments do not  Transition period is especially challenging for

data accessibility

11

Drawdowns

 “Logframe” approaches dominate  Assumptions

 Relatively fixed objectives  Statistical reasoning

12

Methodological stovepipes

13

Increasing collaboration in operations assessment

Pre-conditions Process Structures Mechanisms Challenges

 What are the relevant organisations?

 National aid and development agencies  Independent evaluation units  International organisations  NGO  Private companies  …

 Focus initially on national government

• rganisations

 Development agencies  Evaluation agencies

14

Collaborative partners?

M & E (OECD DAC Definition)

 Evaluation: The systematic and objective

assessment of an on-going or completed project, programme or policy, its design, implementation and results.

 Monitoring: A continuing function that uses

systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing development intervention with indications of the extent of progress and achievement of objectives andprogress in the use of allocated funds.

15

16

Collaborative Framework

Ansell, C.& Gash, A. (2007). Collaborative governance in theory and in practice. Journal of Public Administration Research and Theory, 18, 543-571.

 Partners in Cooperation

 Understand who should be cooperating?

 Spectrum of Interaction

 Understand that cooperation is a spectrum

 Organisational level of interaction

 From tactical to strategic?

 Circumstances of Partnerships

 Out of mission or in mission?

Can real cooperation be “engineered?”….

17

Key Principles

 Understand the Type of Evaluation  Develop Networks  Formalise Agreements  Designing Cooperative Assessments  Mission Coordination  Burden sharing  Consider Other Assessment Processes 18

Mechanisms

Process

19

Tactical HQs JF HQ SHAPE HQ NATO HQ ASSESSREPs Campaign MOE Strategic MOE ASSESSREPs Campaign MOE Strategic MOE PMR Organisations Pol-Mil Review Sprt to Plans & Ops Products Focus End-state and political

• bjectives

Strategic Military Objectives / Effects Operational Military Objectives / Effects Tactical objectives / mission / tasks

 Data exchanges  Survey questions and other data collection

instruments

 Data sources  Final Reports  Staff  Lessons learned, best practises, methods, and

terminology

20

Content of Collaboration

21

Conclusions and Way Ahead

Cooperative Assessment Benefits

 Operational Benefits:

 Data exchange and multiple perspectives  Access to expertise  Reducing duplication of resources  Improving quality and credibility of results

 Political benefits:

 Adherence to international standards  Political will to demonstrate effectiveness of interventions  Coherence in international community

 Improving Cooperation:

 Increasing interaction and dialogue outside of missions

 Organisational Learning:

 Exchange of techniques and best practices

22

Challenges

23  “Transaction” costs of collaboration

 Extra resources  Legal / administrative factors  Travel  Networking

 Analysis process challenges

 Agreeing on scope, objectives and format  Methodology  Team compositions  Timelines

 Political sensitivities

 Controversial findings  Reduction in sensitivity

 Campaign assessment = program evaluation  Ops researchers / analysts as social scientists?  Development of models, theories and

frameworks + research

 Cooperative assessment changes nature of

analysis activity

24

Thoughts…

 Development of NATO concept

 Cooperative Assessment with External Actors  October 2012 release to NATO Nations

 Next version of the NATO Operations

Assessment Handbook

 Update to the NATO Comprehensive

Operations Planning Directive

 Pilot Operations Assessment training course

 Latvian Defence College, Riga, 12 – 16 Nov 2012

25

Future work

26