Evaluating the RGPO Reorganization & the Impact on CBCRP - - PowerPoint PPT Presentation

▶
evaluating the rgpo reorganization the impact on cbcrp
SMART_READER_LITE
LIVE PREVIEW

Evaluating the RGPO Reorganization & the Impact on CBCRP - - PowerPoint PPT Presentation

Sep 18, 2023 360 likes •764 views

Evaluating the RGPO Reorganization & the Impact on CBCRP Program Goals Final Report Presentation September 28, 2012 Janna Cordeiro, MPH Evaluation Consultant 9/28/2012 20011/20012 CBCRP Evaluation Committee Report 1 Outline Background

slide-1
SLIDE 1

Evaluating the RGPO Reorganization & the Impact

  • n CBCRP Program Goals

Final Report Presentation September 28, 2012

Janna Cordeiro, MPH Evaluation Consultant

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 1

slide-2
SLIDE 2

Outline

  • Background
  • Approach and Limitations
  • Evaluation Questions
  • Conclusions
  • Evaluation Studies:

– Summary Results of Studies Reported Previously – Detailed Results of Studies Not Yet Reported

  • Recommendations
  • Acknowledgements

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 2

slide-3
SLIDE 3

Background

  • UC Regents tasked UCOP to streamline operations
  • Cut FTEs/Voluntary Leave Program and Layoffs
  • Consolidation of grant making programs housed in UCOP

UCOP Restructuring Begins 2008

  • Program staff moved to central units
  • Reassignment of staff
  • End of 2011: Evaluation and Dissemination unit eliminated

due to further UCOP budget cuts, resulting in layoffs in January 2012

RGPO Reorganization February 2011

  • Special Research Programs: CBCRP, TRDRP and CHRP
  • UC Grants: Multicampus Research Programs and Initiatives

and Lab Fees Research Program

  • Central units: Contracts and Grants / GBFA
  • Communications and web services contracted with UCOP

Communications and ITS

RGPO Today

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 3

slide-4
SLIDE 4

Preserving the Mission of CBCRP

  • “The Program is a great asset to the state…CBCRP

has …contributed to people’s careers and funded research in areas that are under-explored. The Program is innovative and fills a gap… It’s a real treasure that has done some great work and has more potential for the future.”

  • -- Experienced CBCRP Reviewer

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 4

slide-5
SLIDE 5

RGPO Reorganization Assurances from RGPO Executive Director

  • CBCRP and RGPO centralized unit staff will work together to

accomplish CBCRP program goals

  • CBCRP staff will have the necessary tools and personnel to carry out

their mission

  • CBCRP will receive the same or better quality services for any

activities/functions that have become centralized

  • CBCRP’s unique identity will be retained
  • Efficiencies will be realized resulting in reduced or identical costs to

the Program

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 5

slide-6
SLIDE 6

Approach

  • Council asked for evaluation to monitor impact of

reorganization

  • Hired external evaluator familiar with the Program
  • Collaborative process between RGPO Executive Director,

Council Evaluation Committee, CBCRP staff, and Evaluator

  • Multi-method, variety of stakeholders, over 18 months

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 6

slide-7
SLIDE 7

Evaluation Questions

  • 1. Under the new RGPO structure (2011), how well

can CBCRP accomplish Program Goals including retaining the unique identity of the Program?

  • 2. Is CBCRP receiving the same or better quality of

services from the centralized units?

  • 3. How is the RGPO reorganization impacting

workflow?

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 7

slide-8
SLIDE 8

Limitations

  • Some results are at one point in time and do not

reflect more recent changes

  • Changes to reorganization: extended evaluation,

made changes based on feedback, moving target

  • Goals set at time of evaluation
  • Doesn’t highlight unexpected achievements

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 8

slide-9
SLIDE 9

OVERALL CONCLUSIONS

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 9

slide-10
SLIDE 10

How well can CBCRP Accomplish Program Goals?

  • Over 80% of goals exceeded, met or partially met
  • Communication goals most impacted
  • Retaining CBCRP’s unique aspects is an ongoing

challenge but all are committed to finding workable solutions

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 10

slide-11
SLIDE 11

Is CBCRP receiving the same or better quality services?

  • Centers of Excellence (COEs) and recent staff

changes have resulted in improved quality and customer service

  • Efficiencies yet to be realized for payment related

services

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 11

slide-12
SLIDE 12

How is the reorganization impacting workflow?

  • COEs helpful
  • Time and energy directed toward

reorganization has diverted CBCRP staff from programmatic focus

  • New systems should improve operations at

RGPO, but implementation is still a year away

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 12

slide-13
SLIDE 13

RESULTS: EVALUATION STUDIES

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 13

slide-14
SLIDE 14

Evaluation Studies

Results Reported Previously

  • Brief overview

 Applicants: Online survey  Reviewers: Online survey and Key Informant interviews  Honoraria payments  Program Goals Assessment New Results

  • More details provided

 Communications: Dissemination and Social Media Tracking  Grant Payments  CBCRP Staff Feedback

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 14

slide-15
SLIDE 15

Applicant Survey Summary

Cycle 17 – February 2011

Online survey

  • 77 responses; 86% response rate
  • Overall, positive feedback about CBCRP and application

process

  • Applicants have not been affected by the reorganization
  • Moving forward, Contracts & Grants (C & G) can

monitor customer service

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 15

slide-16
SLIDE 16

Reviewer Surveys Summary

Cycle 17--spring 2011

1. Online survey

  • 37 responses -- 71% response rate

2. Key Informant interviews

  • 9 experienced reviewers
  • Almost all positive feedback: 70% strongly agreed they were

satisfied with experience and remaining 30% agreed.

  • Eliminate video conferencing for review committees
  • C & G continue annual reviewer surveys

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 16

slide-17
SLIDE 17

Honoraria Payments Summary

2008 N=102 2011 N=56 Median Number of Days 23 40 Mean Number of Days 24 43 Range 19-78 33-69 % of Reviewers who did not receive payment by program goal of 42 days 2% 41%

  • In 2011, RGPO able to meet Program goal of 42 days
  • Median higher in 2011 than 2008 and over 40% did not receive payment

by Program Goal of 42 days

  • Grants Budget Finance & Administration (GBFA) can now monitor

honoraria payments

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 17

slide-18
SLIDE 18

Program Goals Assessment

Methods

  • Used 2010-2011 Program Plan adopted by Council
  • Set 72 specific goals within 8 priority areas

Results

  • 83% of CBCRP Program goals were exceeded, met or partially met
  • Unmet goals largely addressed
  • Primary barriers were elimination of Evaluation and

Dissemination unit and pressures on CBCRP Director’s time

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 18

slide-19
SLIDE 19

Program Goals

Recommendations

  • Accountability and responsibility for grant review,

payments and management can now shift from the CBCRP Director to the Directors of central units.

  • CBCRP staff priorities can now change from a focus on

implementing the reorganization to attending to Program- specific initiatives and their continuing scientific and career development.

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 19

slide-20
SLIDE 20

New Results

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 20

slide-21
SLIDE 21

Communications

Systematic survey of dissemination and social media activities Conducted monthly (2/11-6/12) QUESTIONS 1. How often are posts made on Facebook and Twitter? 2. How well is the CBCRP website updated and maintained? 3. Is e-News sent out monthly? 4. Which planned print publications were/were not produced?

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 21

slide-22
SLIDE 22

Communications

Successes

  • Broader reach and more visibility than the past

– National Coalition for Cancer Survivorship (NCCS) award announced through press releases, Twitter, feature stories – Tax check-off widely advertised, including to all UC employees

  • Opportunity to work with UCOP Communications and an
  • utstanding external contractor for social media

– New tax check-off campaign

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 22

slide-23
SLIDE 23

Communications

Facebook: Trend toward improvement

  • Infrequent between 2/11-6/11
  • 6/11 CBCRP took over –more frequent posts from 6/11-10/11
  • No posts in Nov-Dec 2011
  • Regular posts again 1/12-6/12– currently 1060 likes

Twitter: Minimal

  • Few posts between 2/11-6/11
  • June 2011-September 2011 regular posts
  • October 2011-June 2012 almost none

CBCRP Website: Often Out-of-date

  • Often out-of-date content on main pages 2/11-6/12
  • System to update website put in place spring 2012
  • Content management system delayed

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 23

slide-24
SLIDE 24

e-News: sent monthly and on-time

  • Except 2/12
  • CBCRP staff now responsible for e-News

Print Publications

  • 1 tax check-off card and 2 outreach flyers for Building

Sustainable Community Based Research Infrastructure to Better Science (CRIBS)

  • Plans to produce new research findings document delayed

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 24

Communications

slide-25
SLIDE 25

Communications

Conclusions

  • CBCRP has not been able to meet all of the communications goals since

the reorganization ---not operating at previous levels

  • UCOP Communications and contractors have produced broader reach
  • Work with UCOP Communications and Information Technology Services

(ITS) is continuing to evolve

Recommendations

  • Monitor services from UCOP units
  • Develop new communications goals and strategic plan
  • Comprehensive review should remain CBCRP priority until goals are

consistently met

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 25

slide-26
SLIDE 26

Grant Payments

Objectives

1. Assess to what extent RGPO staff were able to meet the program goal of making initial and continuing grant payments within 42 days of the Program Officers releasing the funds. 2. Compare the financial payment services CBCRP received from the SRP finance staff prior to the RGPO reorganization to the services provided by the RGPO central units.

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 26

slide-27
SLIDE 27

Comparison of: Initial Grant Payments

Cycle 17 Cycle 13 # of grants 28 42 Median 46 48 Mean 57 69 Range 9-172 12-312 % met goal 43% 43%

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 27

slide-28
SLIDE 28

Comparison of: Continuing Grant Payments

FY 11-12 FY 07-08 # of payments 45 75 Median 47 49 Mean 75 66 Range 14-289 12-286 % met goal 44% 45%

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 28

slide-29
SLIDE 29

Grant Payments

Conclusions

1. During FY 11-12/Cycle 17, RGPO staff were partially able to meet the Program goal of 6 weeks to payment 2. CBCRP received about the same payment services in 2011 as in 2008

Recommendations

  • Share and discuss with RGPO Executive Director
  • GBFA can now conduct this analysis for Cycle 18 since

processes are now more established

  • New electronic grants management system may streamline

procedures and should provide process reports

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 29

slide-30
SLIDE 30

CBCRP Director and Staff Feedback

  • Online surveys to gather real-time feedback

– Monthly (2/11-5/12)

  • Consensus focus group to gather big picture and

constructive feedback for moving forward

– Held in August 2012 – Summary of all results given to staff ahead of time

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 30

slide-31
SLIDE 31

CBCRP Director and Staff Feedback

Strengths of the Reorganization

  • Upward trend in last 12 months
  • Quality and customer service
  • Leadership in centralized units
  • Centers of Excellence
  • 2012 review committees
  • Outstanding external contractor-Ninth Floor (UCOP

Communications)

  • Broader reach with communications, e.g. Tax check-off

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 31

slide-32
SLIDE 32

CBCRP Director and Staff Feedback

Current Challenges

  • Demands on staff time (Central vs Programmatic responsibilities)
  • Lack of clarity about roles, responsibilities, processes
  • Staff instability in Contracts & Grants unit
  • Delayed systems improvements: electronic grant management

system and website content management system

  • Lack of clarity – UCOP Communications
  • Shift in accountability and responsibility
  • Staff morale

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 32

slide-33
SLIDE 33

CBCRP Director and Staff Feedback

Recommendations

  • Continue with improvements in quality and service
  • Continue COEs
  • Transfer ownership, oversight, and evaluation of processes to

leadership of central units

  • Develop milestones for key processes and track regularly for

continuous quality improvement

  • Recognize CBCRP’s unique needs (legislatively mandated)
  • Design and implement new electronic grants management

system

  • Monitor services from UCOP communications

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 33

slide-34
SLIDE 34

CONCLUSIONS AND RECOMMENDATIONS

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 34

slide-35
SLIDE 35

CONCLUSIONS

  • 1. How well can CBCRP Accomplish Program Goals?

– Over 80% of goals exceeded, met or partially met

  • 2. Is CBCRP receiving the same or better quality

services?

– Improved quality and customer service; efficiencies yet to be realized for payment related services

  • 3. How is the reorganization impacting workflow?

– Positive and negative impacts; new systems likely to produce additional improvements

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 35

slide-36
SLIDE 36

Recommendations

Communications

  • Staff need clarity on services available from UCOP

communications

  • Working with UCOP communications and the Council,

CBCRP staff should revise communications goals, develop a strategic plan, and pursue all available resources to meeting goals

  • Comprehensive review should remain a priority until goals

are consistently met

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 36

slide-37
SLIDE 37

Recommendations

CBCRP Staff Priorities

  • CBCRP Director should allocate more of her time to

developing strategic alliances and launching new initiatives

  • Senior staff can develop and implement new initiatives and

develop scientific expertise

  • The Director and staff will continue to participate in

necessary reorganization activities and COEs

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 37

slide-38
SLIDE 38

Recommendations

Centers of Excellence

  • COEs streamline processes, promote cross-functional

teamwork, show promise towards realizing goals of reorganization

Monitoring Progress

  • Central units can now take lead on evaluating process outcomes

– Reducing time to honoraria and grant payments – Customer service: Reviewer surveys and applicant surveys

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 38

slide-39
SLIDE 39

Recommendations

Accountability

  • Shift accountability for grant review,

payments, and management to Directors of central units

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 39

slide-40
SLIDE 40

Acknowledgements

  • RGPO Executive Director
  • CBCRP Staff
  • Council Evaluation and Priority-Setting Committee

– Terri Burgess (chair), Jeanne Rizzo, Susan Braun, Donna Sanderson, Lisa Barcellos, Kathy Kamath, Cynthia Gomez, and Jon Greif

  • CBCRP Cycle 17 applicants and reviewers
  • GBFA staff for collaboration with retrieving honoraria and

payment data

9/28/2012 20011/20012 CBCRP Evaluation Committee Report 40