Measure and Manage Performance Performance measurement and - - PowerPoint PPT Presentation

How At-Risk Youth and Eldercare Non-Profits Measure and Manage Performance

Recommendations Research Findings Methodology Literature Review Introduction

• Performance measurement and management can be used to ascertain

, , and .

• Stakeholders clamour for evidence of performance and impact.
• Dearth of academic research in Singapore.
• , through a grounded theory approach:
• Semi-structured, first-hand interviews with 16 non-profit organisations.
• Brief analyses of annual reports and financial statements.

Recommendations Research Findings Methodology Literature Review

• Performance measurement and management (

): “specifying which goals to achieve, allocating decisions rights, and measuring and evaluating performance” (Verbeeten, 2006, 428):

• Performance

: collection of information and data.

• Performance

: usage and assessment.

• Non-profit organisations (

) working in the sectors of:

• Of

(15 to 29 years old); and

• In

(65 years old and above).

Recommendations Research Findings Methodology Literature Review

1. How do NPOs in Singapore measure and manage their performance, and how do they use to evaluate their impact? 2. How do NPOs in Singapore , and how do their vary – if they do vary – when compared to broader, academic definitions? 3. Cognisant that beneficiaries – served by dissimilar

• rganisations working in their respective sectors – may

require different services, ? And in what forms?

Recommendations Research Findings Methodology Literature Review

4. To what extent does PMM guide the and

• f these NPOs? In addition, how effective has

the employment of PMM been? 5. What are some of the that NPOs and their employees have faced, in the implementation of PMM within the organisations? Are there in the country which facilitate

• r the

sharing of , and if so how constructive have these platforms been?

Recommendations Research Findings Methodology Literature Review

Recommendations Research Findings Methodology Introduction

• Widespread implementation of PMM in the United States:
• Government Performance and Results Act (

) in 1993.

• GPRA Modernisation Act in 2010.
• Stronger emphasis on data-driven performance targets, evidence-based

evaluations, as well as strategic improvement plans.

• The balanced scorecard and the social return on investment.
• Charity watchdog

:

• Financial health;
• Transparency and accountability; and
• Results reporting.

Recommendations Research Findings Methodology Introduction

• Improves the effectiveness and

efficiency of organisations (Verbeeten, 2008), Kaplan (2001), Olsen (2010).

• Greater accountability to stakeholders

(Moxham and Boaden, 2007), (Poocharoen and Abdullah, 2014), (Zimmerman and Stevens, 2006).

• PMM and its utility are specific to the

NPO (Forbes, 1998), (Speckbacher, 2003).

• Comparisons of performance might

not be consistent, even within similar sectors (Moxham and Boaden, 2007), (Wainwright, 2003), (Schorr, 2016).

• Hard to implement PMM rigorously, to

balance service delivery (Moxham and Boaden, 2007), (Wainwright, 2003), (van Iwaarden et al., 2009).

• Perverse incentives could be created

(Glynn and Murphy, 1996), (Holley et al., 2016), (Verbeeten, 2008).

Recommendations Research Findings Methodology Introduction

“[Although] concerns may be triggered or exacerbated by scandals, the greater issue for charities has been the broader one

• f

and versus and ”.

Gerard Ee (2010), Chairman of the Charity Council

“To maintain and strengthen public trust and confidence, charities have to strive to be , , and ”.

Lawrence Wong (2014), then Acting Minister of the Ministry of Culture, Community, and Youth (MCCY)

Recommendations Research Findings Methodology Introduction

• The MCCY maintains a

.

• Annual reports and

statements of accounts.

• Standard governance

evaluation checklist.

• NPOs publish information
• n their websites.
• :
• Commissioner of

Charities.

• Sector administrators.
• Charity Transparency

Framework.

• “Accountability and

versus financial wrongdoing and regulatory compliance” (Gerard Ee, 2010).

• “Transparent, accountable,

and ” (Lawrence Wong, 2014).

Recommendations Research Findings Methodology Introduction

• in Singapore have

evaluated the use of PMM by NPOs.

• Poocharoen and Abdullah (2014): “in-depth

interviews” with 12 NPOs.

• bin Md.Som et al. (2012): surveys and in-

depth interviews with 60 NPOs.

• “The area of performance measurement is complex and more studies are

needed” (bin Md.Som et al., 2012, 25).

Recommendations Research Findings Methodology Introduction

• National Volunteer and Philanthropy Centre, 13 charity reports published:
• , finance, leadership or governance, and organisation.
• f the NPOs.
• (bin Md.Som et al., 2012):
• Clients’ satisfaction on programmes and

services.

• Efficiency of programmes and services.
• Increase in clients.
• Quality of programmes and services.
• Evaluation of impact.

Recommendations Research Findings Literature Review Introduction

• :
• 12 worked with at-risk youth, 11 worked in eldercare (seven in both).
• All 16 NPOs are charities and IPCs.
• 15 of 16 are administered by the Ministry of Social and Family

Development, and 13 of 16 are registered as socieities.

• :
• 11 (68.8 per cent) were EDs or CEOs.
• The rest were senior managers, directors, or division heads.
• Six of the 16 NPOs are large charities (with more than S$10 million in total

annual receipts). There are no small charities in the sample.

Recommendations Research Findings Literature Review Introduction

• Conduct of

: organise and prepare the data;

• Code and describe the data through

;

• Connect inter-related data through

; and

• Craft

, based on the research questions.

Recommendations Methodology Literature Review Introduction

How do NPOs in Singapore understand PMM, and how do their interpretations vary – if they do vary – when compared to broader, academic definitions? ( ) How do NPOs in Singapore measure and manage their performance, and how do they use data and information to evaluate their impact? ( ) To what extent does PMM guide the

• bjectives and activities
• f these NPOs? In

addition, how effective has the employment of PMM been? ( ) What are some of the challenges that NPOs and their employees have faced, in the implementation of PMM within the organisations? Are there platforms in the country which facilitate training and development or the sharing of best practices, and if so how constructive have these platforms been? ( ) Cognisant that beneficiaries – served by dissimilar

• rganisations working in

their respective sectors – may require different services, does the implementation of PMM vary across these NPOs? And in what forms? ( )

Recommendations Methodology Literature Review Introduction

Mission Vision Objectives

• r Core

Values

Mission, Vision, and Objectives

Programmes and Activities Inputs Outcomes Outputs

Programmes (The Theory of Change and the Logic Model)

The Non-Profit Organisation NPOs in Singapore Funders and Funding Agencies The Government

• f Singapore

Public-at- Large Other Stakeholders Direct Beneficiaries and Immediate Stakeholders Direct Beneficiaries and Immediate Stakeholders Needs Assessment Impact Analysis Reviews Other Programmes Best Practices Controlled for Financial Expenditure

Recommendations Methodology Literature Review Introduction

• Missions and visions are closely aligned with

and .

• Missions and visions are refreshed on a regular basis (12 of

16 have refreshed statements), for different purposes.

• Because

is the main input for NPOs, are emphasised.

• NPOs operate at the programme-level.

Recommendations Methodology Literature Review Introduction

Mission Vision Objectives

• r Core

Values

Mission, Vision, and Objectives

Programmes and Activities Inputs Outcomes Outputs

Programmes (The Theory of Change and the Logic Model)

The Non-Profit Organisation NPOs in Singapore Funders and Funding Agencies The Government

• f Singapore

Public-at- Large Other Stakeholders Direct Beneficiaries and Immediate Stakeholders Direct Beneficiaries and Immediate Stakeholders Needs Assessment Impact Analysis Reviews Other Programmes Best Practices Controlled for Financial Expenditure

Recommendations Methodology Literature Review Introduction

• NPOs have no trouble measuring and managing

and (bin Md.Som et al., 2012).

• Clients’ satisfaction on programmes and services.
• Efficiency of programmes and services.
• Increase in clients.
• Quality of programmes and services.
• PMM is done through

(feedbacks and surveys or key indicators) and (casework, group sessions, or anecdotes) methods.

Recommendations Methodology Literature Review Introduction

• But outputs, outcomes, and impact –

– are much tougher for NPOs.

• Impact measurement are not deep enough, and they

have to be more : “In terms of evaluation, it will be kind of like the first-level … it’s not very deep [because there are too many variables to control for” (1D).

Recommendations Methodology Literature Review Introduction

• But outputs, outcomes, and impact –

– are much tougher for NPOs.

• When long-term sustainability is evaluated, deriving the

for research can be challenging. “When it is in the field, where the real-life human actions are happening … you can’t control everything” (1G).

• Even if impact evaluations are conducted, there is heavy

for data.

• Indicators used may be too

.

Recommendations Methodology Literature Review Introduction

Mission Vision Objectives

• r Core

Values

Mission, Vision, and Objectives

Programmes and Activities Inputs Outcomes Outputs

Programmes (The Theory of Change and the Logic Model)

The Non-Profit Organisation NPOs in Singapore Funders and Funding Agencies The Government

• f Singapore

Public-at- Large Other Stakeholders Direct Beneficiaries and Immediate Stakeholders Direct Beneficiaries and Immediate Stakeholders Needs Assessment Impact Analysis Reviews Other Programmes Best Practices Controlled for Financial Expenditure

Recommendations Methodology Literature Review Introduction

• Needs assessment or analysis is prepared using

and sources.

• are conducted – on a regular basis – for internal

and external stakeholders (all but two NPOs conducted reviews for programmes). “We need not wait until a year to do a review, because as it is ongoing, [for instance], we start to see a lot of clients now coming in with a specific illness or a specific issue, then we are able to then organise training to tackle those issues … We don’t wait for three

• months. It’s just ongoing. As we look at these cases, then we organise training to be

able to empower the counsellors to work more effectively” (2B).

Recommendations Methodology Literature Review Introduction

• There are

between PMM implemented for the programmes and the stakeholders. “In a way we were led by the NCSS for outcome management, so the logic model is a must. It’s a must. So the theory of change comes as to how [and] why our programmes fit in … If our inputs and outputs are not right, and there is no way to change that, or we find it difficult to change, then we will not continue” (1D). “We focus either on output and outcome. But outcome is the one we are really more interested in than the output, so output will involve us doing a proper performance evaluation process, logic modelling, or something like that. So we make sure that everything we do would be able to tell us … how far [we have] achieved, whether it is a good programme, whether it is a not-so-good programme” (1K).

Recommendations Methodology Literature Review Introduction

Mission Vision Objectives

• r Core

Values

Mission, Vision, and Objectives

Programmes and Activities Inputs Outcomes Outputs

Programmes (The Theory of Change and the Logic Model)

The Non-Profit Organisation NPOs in Singapore Funders and Funding Agencies The Government

• f Singapore

Public-at- Large Other Stakeholders Direct Beneficiaries and Immediate Stakeholders Direct Beneficiaries and Immediate Stakeholders Needs Assessment Impact Analysis Reviews Other Programmes Best Practices Controlled for Financial Expenditure

Recommendations Methodology Literature Review Introduction

• Changing circumstances make it hard to define needs.

“[For the elderly], their language capability, their education level, even financial resources, [are] also changing … [So some] tools might be obsolete, because of a different group” (2F) “Needs analysis should not be conducted just based on experience, or the intuition of the social workers or [the] staff members, but has to be more empirical, more systematic, and more scientific” (1B). “[We] require more intensive and extensive market research as to what some of the trends and developments are, [and] what is the monster we are trying to tackle” (1F / 2E).

Recommendations Methodology Literature Review Introduction

• When programmes are reviewed, collecting outcome data

can be difficult. “If you want to measure [a] programme, you are assuming that the outputs of each session [are] the same, the quality is the same, but sometimes in reality there are a lot of variables involved, be it the worker or the clients for that session” (1E / 2D). “You need to find the right indicators [for] research, and I think the language and the cultural barrier is something we need to overcome” (1I / 2H).

Recommendations Methodology Literature Review Introduction

• Funding for research and PMM are hard to come by.

“Robust means the methodology applied, because a lot of the programme evaluation [is] actually not done by trained people. They are just done by social workers who are part-timing programme evaluation” (1K) (emphasis mine).

• Likewise, few NPOs (five) have a research unit.

“Because we are a small organisation, and I think this area is something that is totally new to us … We don’t think an organisation of our size will have a formal unit like that, but rather it would be wiser for us to collaborate with others who have this expertise” (1F / 2E).

Recommendations Methodology Literature Review Introduction

• Measuring and managing long-term outcomes and impact is,

by far, the most difficult. “[A more solid, rigorous research process that goes over the long-term could perhaps be] tied up with an academic institution like NUS, coming in and supplying us [with inputs] as and when … The other [proposal] is the provision of some kind of support, that allows us to track at least five years, you know, the improvements or whatever, and then analyse this, to give us a look, feedback. But that takes time. So through these years … maybe 15,000 people have been impacted, but we don’t know where these 15,000 are, except for having some Facebook accounts. We do have an alumni [group], we try to raise [awareness] and get people through, but this hasn’t been [as solid as I want]” (1D).

Recommendations Methodology Literature Review Introduction

• Measuring and managing long-term outcomes and impact is,

by far, the most difficult. “Happiness of the elderly is really hard to measure. In general I think they say you can be [a] happy nation, there’s a happiness index and things like that, but I think that’s too broad, and you can’t really … localise or contextualise that kind of

• index. And will it be meaningful to use the happiness index for a funding programme?

[Because] happiness is really hard to specify as well” (2F).

Recommendations Methodology Literature Review Introduction

• Measuring and managing long-term outcomes and impact is,

by far, the most difficult. “If we want to go deeper, then we need dedicated and specialised manpower, like researchers. And that bit is a very practical

• concern. [But] not every programme needs that, because you basically need a good

performance measurement system that is practical, that is routinised, and that gives you the data that you want, that cuts across the board … And these are the questions that will not just benefit at the programme-level, but at the sector-level, because then you can go out there and share very confidently that this programme achieved this impact, and I can confidently tell you that this impact is because of this programme” (1G).

Recommendations Methodology Literature Review Introduction

• Better alignment is needed with the government, funders.

“We need to please our funders, to say that you need to hit this at centre-level … So we need to see how to make that and drive together. At the same time, a lot of agencies – I would say – would struggle with quality. So how do we then measure the quality of the service? So for us, what we do is we have an

• ngoing discussion with our funders, co-funders” (1A / 2C).

Recommendations Methodology Literature Review Introduction

• Better alignment is needed with the government, funders.

“Currently everybody seem[s] to be fine, partly because nobody really at the very high level [has looked] into performance measurement as a key factor to drive the social worker sector … [In the] social work sector, a lot of measurements are not done on the basis that you need to depend on it to grow, so it’s like good to have, but don’t have never mind. Because ultimately I feel funding for the VWOs should come more and more not just from the government but from the private sector. So if you don’t have a very robust performance measurement platform, most donors will not want to come in and fund you … Because we just can’t keep depending

• n government coffers for money” (1K).

Recommendations Methodology Literature Review Introduction

Mission Vision Objectives

• r Core

Values

Mission, Vision, and Objectives

Programmes and Activities Inputs Outcomes Outputs

Programmes (The Theory of Change and the Logic Model)

The Non-Profit Organisation NPOs in Singapore Funders and Funding Agencies The Government

• f Singapore

Public-at- Large Other Stakeholders Direct Beneficiaries and Immediate Stakeholders Direct Beneficiaries and Immediate Stakeholders Needs Assessment Impact Analysis Reviews Other Programmes Best Practices Controlled for Financial Expenditure

Recommendations Methodology Literature Review Introduction

• can influence PMM.

“We are trying to look out for partners, government agencies that can fund according to what we need … We tell them what we want to do, they agree, they fund us, so that [gives] us a certain sense of autonomy” (2B).

• Fundraising is more challenging for

. “If the programme is getting big, in terms of the scale and outreach, then we will have to constantly think about sustainability, we have to think about who is going to support this programme in the long run” (1E / 2D).

• No substantial difference in use of PMM, across sectors.

Recommendations Methodology Literature Review Introduction

• Missions and

visions, which are reviewed regularly, are basis for PMM.

• PMM is

employed at the level of programmes.

• Measuring and

managing

• utputs,
• utcomes, and

impact – over the long-term – are tough.

• Needs are

assessed.

• Reviews are

conducted.

• There are

strong links between PMM and the programmes.

• Needs analysis

and data collection.

• Scarce funding

for research (units).

• Better

alignment with stakeholders.

• Best practices.
• Government

funding can influence PMM.

• No substantial

difference across sectors.

Research Findings Methodology Literature Review Introduction

• Promote intra-sector collaboration.
• Rigorous but reasonable indicators.
• Identify research needs and areas.
• Begin with a programme in mind.
• Fund research or research units.
• Encourage pilots of small-scale

research projects.

• Facilitate research, training and

development support from universities.

• Encourage best research practices.
• Demonstrate reciprocity.
• Pay attention to smaller NPOs.
• Assure Singaporeans that NPOs are

transparent and accountable, working to improve results reporting.

• Raise awareness of platforms which

provide information about NPOs.

Research Findings Methodology Literature Review Introduction

• Convenience sampling.
• Research methodology.
• Insights from the stakeholders.
• Beyond the logic models and theories of change.

Research Findings Methodology Literature Review Introduction

• Associate Professor Albert Teo Chu-Ying, National University of Singapore;
• Miss Ang Bee Lian, Ministry of Social and Family Development;
• The 16 NPOs who agreed to the interviews; and
• Everyone who has lent support and encouragement throughout this journey.

How At-Risk Youth and Eldercare Non-Profits Measure and Manage Performance