DEVELOPMENT OF IMPACT MEASURES FOR E-INFRASTRUCTURES M u n i c h , - - PowerPoint PPT Presentation

development of impact measures for e infrastructures
SMART_READER_LITE
LIVE PREVIEW

DEVELOPMENT OF IMPACT MEASURES FOR E-INFRASTRUCTURES M u n i c h , - - PowerPoint PPT Presentation

DEVELOPMENT OF IMPACT MEASURES FOR E-INFRASTRUCTURES M u n i c h , M a r c h 2 9 t h 2 0 1 2 Objectives and key research questions Objectives developing and testing a robust framework for monitoring and evaluation based on information


slide-1
SLIDE 1

M u n i c h , M a r c h 2 9 t h 2 0 1 2

DEVELOPMENT OF IMPACT MEASURES FOR E-INFRASTRUCTURES

slide-2
SLIDE 2

Objectives and key research questions

Objectives

  • developing and testing a robust framework for monitoring and evaluation

based on information obtainable by the projects themselves.

  • analysing socio-economic impact of e-Infrastructures and contributions to EU

policy aims based on the results of the test with a selected number of projects.

  • recommend a set of concrete actions to be taken at the European and

Member States level to implement the resulting monitoring and evaluation system Key research questions

  • Did the program achieve its goal of enabling e-Science?
  • Are there any wider socio-economic impacts in relation to the goals?
slide-3
SLIDE 3
  • Heterogeneity of projects
  • addressed by the development of a typology of e-Infrastructure projects and their

clustering in a multi-criteria matrix.

  • Problem of data availability
  • addressed by a survey and by access to project proposals and any kinds of

documents provided by the project coordinators and/or the Commission.

  • Measuring indirect impacts
  • tackled by consolidation of existing impact assessment approaches and an

intervention logic chart for the relations between objectives, input, output and

  • utcome of the program.
  • Lack of conceptual framework
  • Addressed by extensive feasibility and test studies

Methodology - Challenges

slide-4
SLIDE 4

Intervention Logic

Impacts

Socio‐Economic Impact Impact on science and innovation system (ERA contribution) e‐infrastructure funding e‐Infrastructure programming

Inputs

e‐Infra. services e‐Infra. research

Activities Outputs Outcomes

Innovative technological solutions Sustainable solutions (tech./econ.) Application

  • f innovative

solutions eyond science skilled researchers/ new know‐ ledge and networks Transforma‐ tion of science system + increased human capital Accessibility Increased usage intensity of knowledge (data and scientific inf.) Increased effiency in scientific work Sustainability Innovative‐ ness Efficiency Transfor‐ mative char. Increase/ Creation of problem solving capacities Enable access to sc. Inf/data for larger public by increase of capacities Contribution to EU policy aims Stable but extensible/ scalable structures that support broad participation

Characteristics

e‐Infra. networking

e‐Infrastructures

slide-5
SLIDE 5

Project selection

  • multi-criteria selection process in order to select representative set of projects
  • ensuring framework will be applicable to the program as a whole
  • 21 out of 29 projects responded
  • dimensions of selection
  • domain (implicit)  4-5 per domain
  • Status  ongoing, nearly finished
  • discipline orientation  strong inter- and multidisciplinary focus
  • size (in financial terms)
  • geographical focus  most consortia consit at least of 5 EU partners, several EU-

Non EU

  • Access  mostly open, application based
  • type of actors  orientation towards research institutions, few private companies
slide-6
SLIDE 6

Structure of the questionnaire

  • Measurement of direct results of the projects in the different dimensions
  • Each dimension reflects goals of intervention logic
  • Accessibility
  • to be easily accessible to the public and to provide large enough capacities
  • Efficiency
  • to offer sufficient problem solving capacities
  • Sustainability
  • to develop sustainable activities and infrastructure solutions
  • Innovativeness
  • to develop Innovative technological solutions
  • Transformative character
  • to produce skilled researchers and new knowledge and networks
  • Additional set of questions capturing complementary aspects
slide-7
SLIDE 7

Innovativeness Application of innovative solutions beyond science Transformative character Increased human + intellectual capital transforming science Accessibility Increased usage intensity of knowledge (data and scientific inf.) Efficiency Increased efficiency in scientific work Sustainability Stable and scalable structures supporting broad participation C1 – Knowledge base

  • f. e. results used for teaching, training in-/outsiders,

MA/PhD supervision

C2 – Transformation of science

  • f. e. new science degree, previously unsolvable

questions, multiple disciplines, research standards

C1 – Patents and innovation

  • f. e. patents announced, user reported innovations,

potential for future innovation

C2 – Pool of knowledge

  • f. e. partners from industry and government, user
  • rigin (industry, university, government)

C1 – Continuation and financing

  • f. e. institutions carry on, financing opportunities

C2 – Cooperation

  • f. e. cooperation with projects from same or different

domain

C1 – Problem solving capacity

  • f. e. new software packages, new services, new

problem solving tools)

C2 – User/Projects benefited

  • f. e. user/Projects achieved earlier to results,

user/projects benefited from e-Infrastructure

C3 – Access beyond science

  • f. e. results available, used for teaching, non-

scientific user

C1 – User base

  • f. e. rate of accepted proposals, requests/downloads,

traffic

C2 – Available resources

  • f. e. available information and resources, level of

utilization, length

0,78 0,51 0,52 0,94 0,77

Value of the compostite indicators in the pilot study

(scale from 0 to 1 with 1 indicating the best performance) C3 – Self-efficiency

Self efficiency assessment of e-Infrastructure

Composite Indicator Overview

slide-8
SLIDE 8

Impact Analysis

Impact Areas:

  • Research Excellence and Innovation:
  • Research quality; dissemination of research results; preservation of scientific knowledge; innovation

performance

  • Human Capital:
  • Opportunities for training, lifelong learning, skills; achievements/improvements of the educational system
  • Economy:
  • Productivity; competition; employment; growth
  • Public Authorities:
  • Performance of public authorities; exploitation of public data
  • Third Countries and International Relations
slide-9
SLIDE 9

Contribution to Policy Aims

EU 2020 strategy - Overview

Outcomes

Innovativeness

Application of innovative solutions beyond science

Transformative Char.

Increased human + intellectual capital transforming science

Accessibility

Increased usage intensity of knowledge (data and scientific inf.)

Efficiency

Increased efficiency in scientific work

Sustainability

Stable and scalable structures supporting broad participation

Primary Impacts

Impacts on research excellence and innovation Impacts on human capital Impacts on public authorities Impacts on third countries/intern. relation Impacts on economy EU 2020 Target 3: Education DAE Action Field Single Digital Market

Action 2, 3, 9

DAE Action Field Standards and Interoperability

Action 23, 24

DAE Action Field Research and Innovation

Action 51, 53, 54

DAE Action Field ICT for societal challenges

cross‐cutting

IU Chapter Delivering ERA IU Chapter Promotion of excellence in education and skills d. IU Chapter Single Innovation Market IU Chapter Promotion of openess and IU Chapter Leveraging policies externally

Policy contribution Targets of EU 2020 strategic

  • bjective smart growth

EU 2020 Target 1: Employ- ment EU 2020 Target 2: R&D and Innovation

mutually interrelated

slide-10
SLIDE 10

General Conclusions

Conclusions regarding key research questions:

  • Applied methodology is suitable to indicate the program achievement in

relation to its goals

  • suitable to determine and assess socio-economic impacts and contributions

to EU policy aims

  • there are limitations
  • due to the lack of time series, benchmarks (negative, positive) and reference values
  • no unintended impacts covered
  • any interventions should not be aimed at optimizing single indicators  mutual

interrelations of indicators with other aspects

  • Based on that we suggest:
  • Implementation of a monitoring system
  • development of tool box for further analysis
slide-11
SLIDE 11
  • Regular monitoring system based on the selected items of the pilot study:
  • 10 items each year: easy obtainable, comparable among projects and data points
  • Items collected:

direct measurable impacts on scientific environment, overall science, public

projects‘ pool of knowledge

projects‘ innovative activity

items picturing the development and growth of projects‘ infrastructure and capacity

  • Implementation as part of the annual reporting  easy calculable for projects
  • Complementing survey in non-regular intervals
  • collect additional information to fill the composite indicators enhance analysis of

impacts and policy contribution

  • implementation within the final reporting of the projects

Recommendation

Monitoring system - Survey

slide-12
SLIDE 12

Thank you for your attention!

slide-13
SLIDE 13

Questionnaire development

  • Survey as the basic tool for collecting data from projects  Basis for

measuring the outcomes

  • development in several loops and close coordination with Commission
  • extensive pre-test with seven projects
  • Géant, EGI, NeXpres, PRACE, EUDAT, OpenAire, i4Life
  • Aims of the pre-test
  • to demonstrate the appropriateness and feasibility of the survey questions
  • To test appropriateness and feasibility of the output indicators for the monitoring

system

  • To identify the best output measures in terms of usefulness for impact assessment

and contribution to policy aims

  • formed valuable input for the final questionnaire deployed
slide-14
SLIDE 14

Outcome Analysis (Limitations)

Potential Limitations:

  • Needs sufficient number of observations
  • Some component items based on few responses
  • Large projects could drive results
  • Values need to be treated with care
  • Indicator sensible to results

Solutions:

  • Repeating the study frequently
  • Building composite indicators over several years
  • Control for large projects or projects in starting phase
slide-15
SLIDE 15
  • experiences during the study confirm results of the review of existing

approaches (f. e. UK e-Science program, NSF cyberinfrastructure program) as well as of existing literature  no one size fits all solution

  • Exploitation of the experiences of the different studies in order to develop tool

box for e-Infrastructure monitoring and assessments

  • Additional instruments could be for example:
  • Inventory (siehe eNventory) supported by general surveys (examples: Survey of

Science and Engineering Research Facilities)  coordination required

  • User surveys: integrating needs and challenges of users to understand impact on

their work

  • Bibliometrical support: enable to measure research excellence
  • To be considered:
  • Not only tools, also how and when to use
  • concertation required  understanding intersections between EU and national level

f.e.

Recommendation

Tool Box