DEVELOPMENT OF IMPACT MEASURES FOR E-INFRASTRUCTURES M u n i c h , - PowerPoint PPT Presentation

DEVELOPMENT OF IMPACT MEASURES FOR E-INFRASTRUCTURES M u n i c h , M a r c h 2 9 t h 2 0 1 2

Objectives and key research questions Objectives  developing and testing a robust framework for monitoring and evaluation based on information obtainable by the projects themselves.  analysing socio-economic impact of e-Infrastructures and contributions to EU policy aims based on the results of the test with a selected number of projects.  recommend a set of concrete actions to be taken at the European and Member States level to implement the resulting monitoring and evaluation system Key research questions  Did the program achieve its goal of enabling e-Science?  Are there any wider socio-economic impacts in relation to the goals?

Methodology - Challenges  Heterogeneity of projects • addressed by the development of a typology of e-Infrastructure projects and their clustering in a multi-criteria matrix.  Problem of data availability • addressed by a survey and by access to project proposals and any kinds of documents provided by the project coordinators and/or the Commission.  Measuring indirect impacts • tackled by consolidation of existing impact assessment approaches and an intervention logic chart for the relations between objectives, input, output and outcome of the program.  Lack of conceptual framework • Addressed by extensive feasibility and test studies

Intervention Logic e ‐ infrastructure e ‐ Infrastructure Inputs funding programming e ‐ Infrastructures e ‐ Infra. e ‐ Infra. Activities services research e ‐ Infra. networking Characteristics Innovative ‐ Transfor ‐ Accessibility Efficiency Sustainability ness mative char. Enable access Increase/ skilled to sc. Inf/data Creation of Sustainable Innovative researchers/ for larger problem solutions technological new know ‐ Outputs public by solving (tech./econ.) solutions ledge and increase of capacities networks capacities Increased Stable but Transforma ‐ usage extensible/ Application tion of Increased intensity of scalable of innovative science effiency in Outcomes knowledge structures solutions system + scientific (data and that support eyond increased work scientific broad science human inf.) participation capital Contribution to Impact on science and innovation Impacts Socio ‐ Economic Impact EU policy aims system (ERA contribution)

Project selection  multi-criteria selection process in order to select representative set of projects  ensuring framework will be applicable to the program as a whole  21 out of 29 projects responded  dimensions of selection domain (implicit)  4-5 per domain • • Status  ongoing, nearly finished • discipline orientation  strong inter- and multidisciplinary focus • size (in financial terms) • geographical focus  most consortia consit at least of 5 EU partners, several EU- Non EU • Access  mostly open, application based • type of actors  orientation towards research institutions, few private companies

Structure of the questionnaire  Measurement of direct results of the projects in the different dimensions  Each dimension reflects goals of intervention logic  Accessibility • to be easily accessible to the public and to provide large enough capacities  Efficiency • to offer sufficient problem solving capacities  Sustainability • to develop sustainable activities and infrastructure solutions  Innovativeness • to develop Innovative technological solutions  Transformative character • to produce skilled researchers and new knowledge and networks  Additional set of questions capturing complementary aspects

Composite Indicator Overview C1 – User base Value of the f. e. rate of accepted proposals, requests/downloads, compostite indicators traffic in the pilot study C2 – Available resources (scale from 0 to 1 f. e. available information and resources, level of utilization, length with 1 indicating the best C3 – Access beyond science performance) f. e. results available, used for teaching, non- scientific user Accessibility Increased usage intensity of 0,78 C1 – Problem solving capacity knowledge (data and scientific f. e. new software packages, new services, new inf.) problem solving tools) C2 – User/Projects benefited Efficiency f. e. user/Projects achieved earlier to results, 0,94 Increased efficiency in scientific user/projects benefited from e-Infrastructure work C3 – Self-efficiency Self efficiency assessment of e-Infrastructure Sustainability 0,52 Stable and scalable structures supporting broad participation C1 – Continuation and financing f. e. institutions carry on, financing opportunities C2 – Cooperation Innovativeness 0,51 f. e. cooperation with projects from same or different Application of innovative domain solutions beyond science C1 – Patents and innovation f. e. patents announced, user reported innovations, Transformative character potential for future innovation 0,77 Increased human + intellectual capital transforming science C2 – Pool of knowledge f. e. partners from industry and government, user origin (industry, university, government) C1 – Knowledge base f. e. results used for teaching, training in-/outsiders, MA/PhD supervision C2 – Transformation of science f. e. new science degree, previously unsolvable questions, multiple disciplines, research standards

Impact Analysis Impact Areas:  Research Excellence and Innovation: - Research quality; dissemination of research results; preservation of scientific knowledge; innovation performance  Human Capital: - Opportunities for training, lifelong learning, skills; achievements/improvements of the educational system  Economy: - Productivity; competition; employment; growth  Public Authorities: - Performance of public authorities; exploitation of public data  Third Countries and International Relations

Contribution to Policy Aims EU 2020 strategy - Overview Outcomes Policy contribution Primary Impacts Targets of EU 2020 strategic objective smart growth DAE Action Field Single Accessibility Digital Market Increased usage Impacts on research Action 2, 3, 9 intensity of knowledge excellence and (data and scientific inf.) innovation DAE Action Field EU 2020 Standards and Target 3: Interoperability Education Action 23, 24 DAE Action Field Efficiency Impacts on economy Research and Innovation Increased efficiency in Action 51, 53, 54 scientific work DAE Action Field ICT for societal challenges EU 2020 cross ‐ cutting Sustainability Impacts on human Target 2: capital R&D and Stable and scalable structures supporting Innovation IU Chapter broad participation Delivering ERA IU Chapter Promotion of excellence Impacts on public Innovativeness in education and skills d. authorities Application of EU 2020 innovative solutions IU Chapter Target 1: beyond science Promotion of openess Employ- and ment IU Chapter Impacts on third Transformative Char. Leveraging policies countries/intern. externally relation Increased human + intellectual capital transforming science IU Chapter mutually interrelated Single Innovation Market

General Conclusions Conclusions regarding key research questions:  Applied methodology is suitable to indicate the program achievement in relation to its goals  suitable to determine and assess socio-economic impacts and contributions to EU policy aims  there are limitations • due to the lack of time series, benchmarks (negative, positive) and reference values • no unintended impacts covered • any interventions should not be aimed at optimizing single indicators  mutual interrelations of indicators with other aspects  Based on that we suggest: • Implementation of a monitoring system • development of tool box for further analysis

Recommendation Monitoring system - Survey  Regular monitoring system based on the selected items of the pilot study: • 10 items each year: easy obtainable, comparable among projects and data points • Items collected:  direct measurable impacts on scientific environment, overall science, public  projects‘ pool of knowledge  projects‘ innovative activity  items picturing the development and growth of projects‘ infrastructure and capacity • Implementation as part of the annual reporting  easy calculable for projects  Complementing survey in non-regular intervals • collect additional information to fill the composite indicators enhance analysis of impacts and policy contribution • implementation within the final reporting of the projects

for your attention! Thank you

Questionnaire development  Survey as the basic tool for collecting data from projects  Basis for measuring the outcomes  development in several loops and close coordination with Commission  extensive pre-test with seven projects • Géant, EGI, NeXpres, PRACE, EUDAT, OpenAire, i4Life  Aims of the pre-test • to demonstrate the appropriateness and feasibility of the survey questions • To test appropriateness and feasibility of the output indicators for the monitoring system • To identify the best output measures in terms of usefulness for impact assessment and contribution to policy aims  formed valuable input for the final questionnaire deployed