Evaluating the RGPO Reorganization & the Impact on CBCRP Program Goals Final Report Presentation September 28, 2012 Janna Cordeiro, MPH Evaluation Consultant 9/28/2012 20011/20012 CBCRP Evaluation Committee Report 1
Outline Background • Approach and Limitations • Evaluation Questions • Conclusions • Evaluation Studies: • – Summary Results of Studies Reported Previously – Detailed Results of Studies Not Yet Reported Recommendations • Acknowledgements • 9/28/2012 20011/20012 CBCRP Evaluation Committee Report 2
Background UCOP • UC Regents tasked UCOP to streamline operations Restructuring Begins • Cut FTEs/Voluntary Leave Program and Layoffs • Consolidation of grant making programs housed in UCOP 2008 • Program staff moved to central units RGPO • Reassignment of staff Reorganization • End of 2011: Evaluation and Dissemination unit eliminated due to further UCOP budget cuts, resulting in layoffs in February 2011 January 2012 • Special Research Programs: CBCRP, TRDRP and CHRP • UC Grants: Multicampus Research Programs and Initiatives and Lab Fees Research Program RGPO Today • Central units: Contracts and Grants / GBFA • Communications and web services contracted with UCOP Communications and ITS 9/28/2012 20011/20012 CBCRP Evaluation Committee Report 3
Preserving the Mission of CBCRP • “The Program is a great asset to the state…CBCRP has …contributed to people’s careers and funded research in areas that are under-explored. The Program is innovative and fills a gap… It’s a real treasure that has done some great work and has more potential for the future.” --- Experienced CBCRP Reviewer 9/28/2012 20011/20012 CBCRP Evaluation Committee Report 4
RGPO Reorganization Assurances from RGPO Executive Director • CBCRP and RGPO centralized unit staff will work together to accomplish CBCRP program goals • CBCRP staff will have the necessary tools and personnel to carry out their mission • CBCRP will receive the same or better quality services for any activities/functions that have become centralized • CBCRP’s unique identity will be retained • Efficiencies will be realized resulting in reduced or identical costs to the Program 9/28/2012 20011/20012 CBCRP Evaluation Committee Report 5
Approach • Council asked for evaluation to monitor impact of reorganization • Hired external evaluator familiar with the Program • Collaborative process between RGPO Executive Director, Council Evaluation Committee, CBCRP staff, and Evaluator • Multi-method, variety of stakeholders, over 18 months 9/28/2012 20011/20012 CBCRP Evaluation Committee Report 6
Evaluation Questions 1. Under the new RGPO structure (2011), how well can CBCRP accomplish Program Goals including retaining the unique identity of the Program? 2. Is CBCRP receiving the same or better quality of services from the centralized units? 3. How is the RGPO reorganization impacting workflow? 9/28/2012 20011/20012 CBCRP Evaluation Committee Report 7
Limitations • Some results are at one point in time and do not reflect more recent changes • Changes to reorganization: extended evaluation, made changes based on feedback, moving target • Goals set at time of evaluation • Doesn’t highlight unexpected achievements 9/28/2012 20011/20012 CBCRP Evaluation Committee Report 8
OVERALL CONCLUSIONS 9/28/2012 20011/20012 CBCRP Evaluation Committee Report 9
How well can CBCRP Accomplish Program Goals? • Over 80% of goals exceeded, met or partially met • Communication goals most impacted • Retaining CBCRP’s unique aspects is an ongoing challenge but all are committed to finding workable solutions 9/28/2012 20011/20012 CBCRP Evaluation Committee Report 10
Is CBCRP receiving the same or better quality services? • Centers of Excellence (COEs) and recent staff changes have resulted in improved quality and customer service • Efficiencies yet to be realized for payment related services 9/28/2012 20011/20012 CBCRP Evaluation Committee Report 11
How is the reorganization impacting workflow? • COEs helpful • Time and energy directed toward reorganization has diverted CBCRP staff from programmatic focus • New systems should improve operations at RGPO, but implementation is still a year away 9/28/2012 20011/20012 CBCRP Evaluation Committee Report 12
RESULTS: EVALUATION STUDIES 9/28/2012 20011/20012 CBCRP Evaluation Committee Report 13
Evaluation Studies Results Reported Previously New Results - More details provided - Brief overview Applicants: Online survey Communications: Reviewers: Online survey Dissemination and Social and Key Informant Media Tracking interviews Grant Payments Honoraria payments CBCRP Staff Feedback Program Goals Assessment 20011/20012 CBCRP Evaluation Committee 9/28/2012 14 Report
Applicant Survey Summary Cycle 17 – February 2011 Online survey • 77 responses; 86% response rate • Overall, positive feedback about CBCRP and application process • Applicants have not been affected by the reorganization Moving forward, Contracts & Grants (C & G) can monitor customer service 9/28/2012 20011/20012 CBCRP Evaluation Committee Report 15
Reviewer Surveys Summary Cycle 17--spring 2011 1. Online survey • 37 responses -- 71% response rate 2. Key Informant interviews 9 experienced reviewers • • Almost all positive feedback: 70% strongly agreed they were satisfied with experience and remaining 30% agreed. Eliminate video conferencing for review committees C & G continue annual reviewer surveys 9/28/2012 20011/20012 CBCRP Evaluation Committee Report 16
Honoraria Payments Summary 2008 N=102 2011 N=56 Median Number of Days 23 40 Mean Number of Days 24 43 Range 19-78 33-69 % of Reviewers who did 2% 41% not receive payment by program goal of 42 days • In 2011, RGPO able to meet Program goal of 42 days • Median higher in 2011 than 2008 and over 40% did not receive payment by Program Goal of 42 days Grants Budget Finance & Administration (GBFA) can now monitor honoraria payments 9/28/2012 20011/20012 CBCRP Evaluation Committee Report 17
Program Goals Assessment Methods • Used 2010-2011 Program Plan adopted by Council • Set 72 specific goals within 8 priority areas Results • 83% of CBCRP Program goals were exceeded, met or partially met • Unmet goals largely addressed • Primary barriers were elimination of Evaluation and Dissemination unit and pressures on CBCRP Director’s time 9/28/2012 20011/20012 CBCRP Evaluation Committee Report 18
Program Goals Recommendations Accountability and responsibility for grant review, payments and management can now shift from the CBCRP Director to the Directors of central units. CBCRP staff priorities can now change from a focus on implementing the reorganization to attending to Program- specific initiatives and their continuing scientific and career development. 9/28/2012 20011/20012 CBCRP Evaluation Committee Report 19
New Results 20011/20012 CBCRP Evaluation Committee 9/28/2012 20 Report
Communications Systematic survey of dissemination and social media activities Conducted monthly (2/11-6/12) QUESTIONS 1. How often are posts made on Facebook and Twitter? 2. How well is the CBCRP website updated and maintained? 3. Is e-News sent out monthly? 4. Which planned print publications were/were not produced? 9/28/2012 20011/20012 CBCRP Evaluation Committee Report 21
Communications Successes • Broader reach and more visibility than the past – National Coalition for Cancer Survivorship (NCCS) award announced through press releases, Twitter, feature stories – Tax check-off widely advertised, including to all UC employees • Opportunity to work with UCOP Communications and an outstanding external contractor for social media – New tax check-off campaign 9/28/2012 20011/20012 CBCRP Evaluation Committee Report 22
Communications Facebook: Trend toward improvement • Infrequent between 2/11-6/11 • 6/11 CBCRP took over –more frequent posts from 6/11-10/11 • No posts in Nov-Dec 2011 • Regular posts again 1/12-6/12– currently 1060 likes Twitter: Minimal • Few posts between 2/11-6/11 • June 2011-September 2011 regular posts • October 2011-June 2012 almost none CBCRP Website: Often Out-of-date • Often out-of-date content on main pages 2/11-6/12 • System to update website put in place spring 2012 • Content management system delayed 9/28/2012 20011/20012 CBCRP Evaluation Committee Report 23
Communications e-News: sent monthly and on-time • Except 2/12 • CBCRP staff now responsible for e-News Print Publications • 1 tax check-off card and 2 outreach flyers for Building Sustainable Community Based Research Infrastructure to Better Science (CRIBS) • Plans to produce new research findings document delayed 9/28/2012 20011/20012 CBCRP Evaluation Committee Report 24
Recommend
More recommend
