DEVELOPMENT OF IMPACT MEASURES FOR E-INFRASTRUCTURES M u n i c h , - - PowerPoint PPT Presentation
DEVELOPMENT OF IMPACT MEASURES FOR E-INFRASTRUCTURES M u n i c h , - - PowerPoint PPT Presentation
DEVELOPMENT OF IMPACT MEASURES FOR E-INFRASTRUCTURES M u n i c h , M a r c h 2 9 t h 2 0 1 2 Objectives and key research questions Objectives developing and testing a robust framework for monitoring and evaluation based on information
Objectives and key research questions
Objectives
- developing and testing a robust framework for monitoring and evaluation
based on information obtainable by the projects themselves.
- analysing socio-economic impact of e-Infrastructures and contributions to EU
policy aims based on the results of the test with a selected number of projects.
- recommend a set of concrete actions to be taken at the European and
Member States level to implement the resulting monitoring and evaluation system Key research questions
- Did the program achieve its goal of enabling e-Science?
- Are there any wider socio-economic impacts in relation to the goals?
- Heterogeneity of projects
- addressed by the development of a typology of e-Infrastructure projects and their
clustering in a multi-criteria matrix.
- Problem of data availability
- addressed by a survey and by access to project proposals and any kinds of
documents provided by the project coordinators and/or the Commission.
- Measuring indirect impacts
- tackled by consolidation of existing impact assessment approaches and an
intervention logic chart for the relations between objectives, input, output and
- utcome of the program.
- Lack of conceptual framework
- Addressed by extensive feasibility and test studies
Methodology - Challenges
Intervention Logic
Impacts
Socio‐Economic Impact Impact on science and innovation system (ERA contribution) e‐infrastructure funding e‐Infrastructure programming
Inputs
e‐Infra. services e‐Infra. research
Activities Outputs Outcomes
Innovative technological solutions Sustainable solutions (tech./econ.) Application
- f innovative
solutions eyond science skilled researchers/ new know‐ ledge and networks Transforma‐ tion of science system + increased human capital Accessibility Increased usage intensity of knowledge (data and scientific inf.) Increased effiency in scientific work Sustainability Innovative‐ ness Efficiency Transfor‐ mative char. Increase/ Creation of problem solving capacities Enable access to sc. Inf/data for larger public by increase of capacities Contribution to EU policy aims Stable but extensible/ scalable structures that support broad participation
Characteristics
e‐Infra. networking
e‐Infrastructures
Project selection
- multi-criteria selection process in order to select representative set of projects
- ensuring framework will be applicable to the program as a whole
- 21 out of 29 projects responded
- dimensions of selection
- domain (implicit) 4-5 per domain
- Status ongoing, nearly finished
- discipline orientation strong inter- and multidisciplinary focus
- size (in financial terms)
- geographical focus most consortia consit at least of 5 EU partners, several EU-
Non EU
- Access mostly open, application based
- type of actors orientation towards research institutions, few private companies
Structure of the questionnaire
- Measurement of direct results of the projects in the different dimensions
- Each dimension reflects goals of intervention logic
- Accessibility
- to be easily accessible to the public and to provide large enough capacities
- Efficiency
- to offer sufficient problem solving capacities
- Sustainability
- to develop sustainable activities and infrastructure solutions
- Innovativeness
- to develop Innovative technological solutions
- Transformative character
- to produce skilled researchers and new knowledge and networks
- Additional set of questions capturing complementary aspects
Innovativeness Application of innovative solutions beyond science Transformative character Increased human + intellectual capital transforming science Accessibility Increased usage intensity of knowledge (data and scientific inf.) Efficiency Increased efficiency in scientific work Sustainability Stable and scalable structures supporting broad participation C1 – Knowledge base
- f. e. results used for teaching, training in-/outsiders,
MA/PhD supervision
C2 – Transformation of science
- f. e. new science degree, previously unsolvable
questions, multiple disciplines, research standards
C1 – Patents and innovation
- f. e. patents announced, user reported innovations,
potential for future innovation
C2 – Pool of knowledge
- f. e. partners from industry and government, user
- rigin (industry, university, government)
C1 – Continuation and financing
- f. e. institutions carry on, financing opportunities
C2 – Cooperation
- f. e. cooperation with projects from same or different
domain
C1 – Problem solving capacity
- f. e. new software packages, new services, new
problem solving tools)
C2 – User/Projects benefited
- f. e. user/Projects achieved earlier to results,
user/projects benefited from e-Infrastructure
C3 – Access beyond science
- f. e. results available, used for teaching, non-
scientific user
C1 – User base
- f. e. rate of accepted proposals, requests/downloads,
traffic
C2 – Available resources
- f. e. available information and resources, level of
utilization, length
0,78 0,51 0,52 0,94 0,77
Value of the compostite indicators in the pilot study
(scale from 0 to 1 with 1 indicating the best performance) C3 – Self-efficiency
Self efficiency assessment of e-Infrastructure
Composite Indicator Overview
Impact Analysis
Impact Areas:
- Research Excellence and Innovation:
- Research quality; dissemination of research results; preservation of scientific knowledge; innovation
performance
- Human Capital:
- Opportunities for training, lifelong learning, skills; achievements/improvements of the educational system
- Economy:
- Productivity; competition; employment; growth
- Public Authorities:
- Performance of public authorities; exploitation of public data
- Third Countries and International Relations
Contribution to Policy Aims
EU 2020 strategy - Overview
Outcomes
Innovativeness
Application of innovative solutions beyond science
Transformative Char.
Increased human + intellectual capital transforming science
Accessibility
Increased usage intensity of knowledge (data and scientific inf.)
Efficiency
Increased efficiency in scientific work
Sustainability
Stable and scalable structures supporting broad participation
Primary Impacts
Impacts on research excellence and innovation Impacts on human capital Impacts on public authorities Impacts on third countries/intern. relation Impacts on economy EU 2020 Target 3: Education DAE Action Field Single Digital Market
Action 2, 3, 9
DAE Action Field Standards and Interoperability
Action 23, 24
DAE Action Field Research and Innovation
Action 51, 53, 54
DAE Action Field ICT for societal challenges
cross‐cutting
IU Chapter Delivering ERA IU Chapter Promotion of excellence in education and skills d. IU Chapter Single Innovation Market IU Chapter Promotion of openess and IU Chapter Leveraging policies externally
Policy contribution Targets of EU 2020 strategic
- bjective smart growth
EU 2020 Target 1: Employ- ment EU 2020 Target 2: R&D and Innovation
mutually interrelated
General Conclusions
Conclusions regarding key research questions:
- Applied methodology is suitable to indicate the program achievement in
relation to its goals
- suitable to determine and assess socio-economic impacts and contributions
to EU policy aims
- there are limitations
- due to the lack of time series, benchmarks (negative, positive) and reference values
- no unintended impacts covered
- any interventions should not be aimed at optimizing single indicators mutual
interrelations of indicators with other aspects
- Based on that we suggest:
- Implementation of a monitoring system
- development of tool box for further analysis
- Regular monitoring system based on the selected items of the pilot study:
- 10 items each year: easy obtainable, comparable among projects and data points
- Items collected:
direct measurable impacts on scientific environment, overall science, public
projects‘ pool of knowledge
projects‘ innovative activity
items picturing the development and growth of projects‘ infrastructure and capacity
- Implementation as part of the annual reporting easy calculable for projects
- Complementing survey in non-regular intervals
- collect additional information to fill the composite indicators enhance analysis of
impacts and policy contribution
- implementation within the final reporting of the projects
Recommendation
Monitoring system - Survey
Thank you for your attention!
Questionnaire development
- Survey as the basic tool for collecting data from projects Basis for
measuring the outcomes
- development in several loops and close coordination with Commission
- extensive pre-test with seven projects
- Géant, EGI, NeXpres, PRACE, EUDAT, OpenAire, i4Life
- Aims of the pre-test
- to demonstrate the appropriateness and feasibility of the survey questions
- To test appropriateness and feasibility of the output indicators for the monitoring
system
- To identify the best output measures in terms of usefulness for impact assessment
and contribution to policy aims
- formed valuable input for the final questionnaire deployed
Outcome Analysis (Limitations)
Potential Limitations:
- Needs sufficient number of observations
- Some component items based on few responses
- Large projects could drive results
- Values need to be treated with care
- Indicator sensible to results
Solutions:
- Repeating the study frequently
- Building composite indicators over several years
- Control for large projects or projects in starting phase
- experiences during the study confirm results of the review of existing
approaches (f. e. UK e-Science program, NSF cyberinfrastructure program) as well as of existing literature no one size fits all solution
- Exploitation of the experiences of the different studies in order to develop tool
box for e-Infrastructure monitoring and assessments
- Additional instruments could be for example:
- Inventory (siehe eNventory) supported by general surveys (examples: Survey of
Science and Engineering Research Facilities) coordination required
- User surveys: integrating needs and challenges of users to understand impact on
their work
- Bibliometrical support: enable to measure research excellence
- To be considered:
- Not only tools, also how and when to use
- concertation required understanding intersections between EU and national level