Multi-organisational collaboration in operations assessment and evaluation: Prospects, challenges, and implications for practice 29 th ISMOR, 2012 Andy Williams Operational Analysis Branch HQ Supreme Allied Commander Transformation The views expressed in this article are the views of the author and do not necessarily represent the views of NATO or any other organisation 1
Allied Command Transformation ACT is NATO’s leading agent for change, driving, facilitating, and advocating continuous improvement of Alliance capabilities to maintain and enhance the military relevance and effectiveness of the Alliance. 2
ACT Trident Interoperability Strategic Thinking Lessons Learned Capability Partners + Development Outreach Lessons Education & Identified Training Coherence 3
Abstract Measuring progress and results of an organisation’s activities —“assessment” in the military and “evaluation” in the civilian sector— are typically arranged as principal-agent relationships between a management, leadership, funding or accountability body, and the evaluating body. Recent practice has seen the emergence of collaboration between assessment and evaluation organisations involving more complex arrangements with either multiple principals or multiple agents, or both. Furthermore, NATO has expressed an ambition for military operations assessment organisations to work more closely with civilian evaluation agencies in conflict or fragile environments. This paper proposes a framework in which military assessment organisations can interact with civilian evaluation departments. The framework considers: the rationale for collaborating on operations assessments; the necessary pre- conditions; the challenges and transaction costs involved; and the collaboration process, including structural issues of governance and administration, mechanisms of interaction, and methodology. Ideas for empirical research and further theoretical development are presented. Keywords: Operations Assessment, Monitoring and Evaluation, Evaluation, Collaboration 4
Operations Assessment Operations Assessment “ the function that enables the measurement of progress and results of operations in a military context , and the subsequent development of conclusions and recommendations that support decision making ” (COPD, p. 5 -1). A.k.a. campaign assessment; operational assessment, assessment Many similarities to “monitoring and evaluation”
Context Assessment Capability Development Plan Concept development Refinement of NATO policy and directives Experimentation Development of new training Analytical support Capture of current problems in assessment Lessons identified / learned Research projects Literature reviews, workshops, interviews 6
Agenda Problems in operations assessment Rationale for increased collaboration Multi-organisational collaboration framework Conclusions and future work 7
Current challenges in Ops Assessment Data limitations Evaluating causal impacts Methodological stovepiping Drawdowns and transition 8
Data limitations Abundance of security-incident related data Allows granular measurement of change Directly relatable to plan elements Abundance of other military data and reports Allow granular measurement Not easily relatable to plan elements Abundance of governance, development, economic, social, regional data Allows retrospective, strategic level assessments Not directly relatable to plan elements Conclusion: militaries are planning for governance and development impacts, but have limited data to assess progress 9
Evaluating causal impacts Current military plans clearly call for evaluation of complex causalities, which isn’t possible without access to relevant non-military data 10
Drawdowns Military missions eventually drawdown Political requirements for assessments do not Transition period is especially challenging for data accessibility 11
Methodological stovepipes “ Logframe ” approaches dominate Assumptions Relatively fixed objectives Statistical reasoning 12
Increasing collaboration in operations assessment Pre-conditions Process Structures Mechanisms Challenges 13
Collaborative partners? What are the relevant organisations? National aid and development agencies Independent evaluation units International organisations NGO Private companies … Focus initially on national government organisations Development agencies Evaluation agencies 14
M & E (OECD DAC Definition) Evaluation : The systematic and objective assessment of an on-going or completed project, programme or policy, its design, implementation and results. Monitoring : A continuing function that uses systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing development intervention with indications of the extent of progress and achievement of objectives andprogress in the use of allocated funds. 15
Collaborative Framework Ansell, C.& Gash, A. (2007). Collaborative governance in theory and in practice. Journal of Public Administration Research and Theory, 18, 543-571. 16
Key Principles Partners in Cooperation Understand who should be cooperating? Spectrum of Interaction Understand that cooperation is a spectrum Organisational level of interaction From tactical to strategic? Circumstances of Partnerships Out of mission or in mission? Can real cooperation be “engineered?”…. 17
Mechanisms Understand the Type of Evaluation Develop Networks Formalise Agreements Designing Cooperative Assessments Mission Coordination Burden sharing Consider Other Assessment Processes 18
Process Organisations Products Focus Pol-Mil Review Sprt to Plans & Ops End-state and political NATO HQ PMR objectives Strategic Military SHAPE HQ Strategic MOE Strategic MOE Objectives / Effects Operational Military JF HQ Campaign MOE Campaign MOE Objectives / Effects Tactical objectives / Tactical HQs ASSESSREPs ASSESSREPs mission / tasks 19
Content of Collaboration Data exchanges Survey questions and other data collection instruments Data sources Final Reports Staff Lessons learned, best practises, methods, and terminology 20
Conclusions and Way Ahead 21
Cooperative Assessment Benefits Operational Benefits: Data exchange and multiple perspectives Access to expertise Reducing duplication of resources Improving quality and credibility of results Political benefits: Adherence to international standards Political will to demonstrate effectiveness of interventions Coherence in international community Improving Cooperation: Increasing interaction and dialogue outside of missions Organisational Learning: Exchange of techniques and best practices 22
Challenges “Transaction” costs of collaboration Extra resources Legal / administrative factors Travel Networking Analysis process challenges Agreeing on scope, objectives and format Methodology Team compositions Timelines Political sensitivities Controversial findings Reduction in sensitivity 23
Thoughts… Campaign assessment = program evaluation Ops researchers / analysts as social scientists? Development of models, theories and frameworks + research Cooperative assessment changes nature of analysis activity 24
Future work Development of NATO concept Cooperative Assessment with External Actors October 2012 release to NATO Nations Next version of the NATO Operations Assessment Handbook Update to the NATO Comprehensive Operations Planning Directive Pilot Operations Assessment training course Latvian Defence College, Riga, 12 – 16 Nov 2012 25
26
Recommend
More recommend
