2019 RSR Town Hall Debrief RYAN WHI TE HIV/AIDS PROGRAM SERVICES - - PDF document

2019 RSR Town Hall Debrief

RYAN WHI TE HIV/AIDS PROGRAM SERVICES REP O RT (RSR) HRSA HI V/AI DS BUREAU JUNE 2 4 , 2 0 2 0

ELLIE Hello everyone. I’m Ellie Coombs from the DART Team. I can see that people are still joining the webinar so we’re going to wait a few minutes before we get started. That gives you time to send that last email or grab more water or coffee. I’ll be back in about two minutes! Welcome to today’s Webcast. Thank you so much for joining us today! My name is Ellie Coombs. I’m a member of the DART Team, one of several groups engaged by HAB to provide training and technical assistance to Ryan White HIV/AIDS Program recipients and subrecipients for the RSR. Today’s Webcast is presented by Jennifer Livingston from Ryan White Data Support, and AJ Jones also from the DART Team. Now this webcast is a little different than the other webcasts we do. It provides an opportunity to reflect on the most recent RSR submission. We’ll start with sharing an

• verview of the submission including both successes as well as some of the challenges that were
• identified. We also want your feedback, suggestions or questions about the 2019 RSR. We’ll use

polls to capture a little information but also hope that you share your feedback and questions during the Q & A period. If you think of feedback or a question during the presentation, you can type it into the question function on the right-hand side of your screen. There will also be an opportunity to ask question live after the presentation. I’ll share how to do that later in the Webcast. Now before we start, I’m going to answer

• ne
• f

the most commonly asked questions about the slides. The recording

• f

today’s webinar will be available

• n

the TargetHIV website within

• ne

week 1

• f

the webinar; the slides and written question and answer are usually available within two weeks. 1

Disclaimer

Today’s webinar is supported by the following

• rganizations,

and the contents are those

• f

the author(s) and do not necessarily represent the

• fficial

views

• f,

nor an endorsement, by the Health Resources and Services Administration (HRSA), U.S. Department

• f

Health and Human Services (HHS),

• r

the U.S. Government.

• CAI

and its partners Abt Associates and Mission Analytics, supported by HRSA, part

• f

HHS as part

• f

an award totaling $450,000.

• Ryan

White HIV/AIDS Program Data Coordination and Technical Assistance contract is supported by HRSA

• f

HHS as part

• f

an award totaling $5,092,875.59.

2

Today’s webinar is supported by the organizations shown on the slide, and the contents are those of the author(s) and do not necessarily represent the official views

• f, nor an endorsement, by the Health Resources Services Administration, the U.S.

Department of Health and Human Services, or the U.S. Government. Now I’d like to turn the webcast over to AJ. 2

Webcast Overview

2019 RSR Submission Statistics RSR Submission Challenges RSR Comment Review Next Steps and Resources

3

AJ Thanks Ellie, today we are going take a look and review the 2019 RSR submission

• period. We’ll start with some ** 2019 RSR submission statistics. Then, we’ll move on

to the ** RSR submission challenges that occurred during this year’s submission period in the Recipient Report, Provider Report, and client-level data.. Next, we’ll take a look at the ** RSR Comment Review currently underway and go over some of the key findings seen so far. And finally we’ll go through the various ** technical assistance resources available to assist you. Let’s get started! 3

2018 RSR Deadline 2019 RSR Deadline March 25, 2019 April 30, 2020 Percent of Recipient Reports Submitted

97.5% 97.5%

by the RSR Deadline Percent of Provider Reports Submitted

99.1% 99.4%

by the RSR Deadline

2019 RSR Data Submission Statistics

4

This season’s submission started like any other but as we all know, the final deadline was extended from March 30 to April 30 to give agencies an extra month to submit in the midst of the COVID-19 pandemic. Thankfully, many agencies were still able to submit by this extended

• deadline. 97.5% of Recipient Reports were in submitted status by April 30, the same amount

that were submitted by last year’s deadline. Additionally, we actually saw an increase in the number of providers that were able to submit by this year’s deadline with 99.4% in submitted status by April 30 as opposed to last year where 99.1% had submitted by the deadline. We are always working towards a 100 percent on-time submission rate so it’s really great to see such high numbers even given the national COVID-19 situation. Congratulations to everyone on another successful submission and thank you for your hard work during a very difficult time. I’m going to turn the presentation over to my colleague Ruchi so we can jump into a poll question to see how the deadline extension affected your agency. Ruchi? 4

Poll Question #1

5

How helpful was the 30 day extension to submit the RSR? Please select only one.

• Very helpful. My organization would not have been able to meet the March 30 deadline for

submission.

• Somewhat helpful. There is a good chance that my agency would have met the original

deadline, but we appreciated having more time.

• No impact since we were on track to meet the original March 30th deadline.

Thanks Ruchi. [contextualize response] 5

Webcast Overview (2)

2019 RSR Submission Statistics RSR Submission Challenges RSR Comment Review Next Steps and Resources

6

Next, we’re going to get into some of the submission challenges that occurred during the 2019 RSR reporting window 6

• EHB/Web System

Recipient Report Provider Report Client level Data

EHB/RSR Web System Issues

• 1. The

HRSA server moved during the submission cycle

• Bookmarked

links no longer worked

• The

first search result linked Providers to the test system

• https://grants6.hrsa.gov/ha

b/RegloginApp/admin/login .aspx?application=rsrApp Make sure to use:

7

AJ Starting off with high-level issues with the RSR web system, we had one major challenge this year: the server that hosts the EHB and the RSR Web System moved to a new server during the submission season. For you veterans that have worked on the RSR before, this meant our archived bookmark for “performance.hrsa.gov” stopped working during the year. While this wasn’t a big issue for recipients, many providers ended up being re-directed to the test environment for the RSR Web System. Some folks uploaded data and submitted in the test system, which did not link up to their actual live 2019 RSR deliverable. Make sure that moving forward you use the link with grants6 (which should be easy to find now that the top result for the test system has been fixed). 7

EHB/Web System Recipient Report Provider Report Client-level Data

EHB/RSR Web System Issues (2)

• 2. Some

archived reports are not

• pening

correctly

• PDFs

version still work as expected

• 3. Minor logic

errors were updated in the RSR Upload Completeness Report

8

AJ Additionally, a provider reported to us that some archived deliverables, such as old Upload Completeness Reports in Excel format, cannot be opened. We’re working to figure out what’s going on with these reports, but you can still access archived PDFs

• f all your previous reports.

Finally, several providers pointed out an issue in the logic used in the Upload Completeness Report, or UCR, for the HIV diagnosis year data elements. Hopefully you are all familiar with the UCR, but for a quick overview this is a built in report in the RSR Web System that reflects the data you upload for each element. This year, the HIV diagnosis year element was incorrectly coded as required for all clients – based on your feedback, we were also able to correct this issue and re-create the UCRs for all providers prior to the final submission. 8

EHB/Web System Recipient Report Provider Report Client-level Data

Recipient Report

• Beginning

this year, agencies could submit client-level data under Eligible Services Reporting

• See

RSR in Focus: Understanding Eligible Services for 2019 Data

• Recipients

were redirected to the incorrect inbox

• Recipient’s

inbox was showing the incorrect recipient

• r

program part

• Recipients

were unable to view their providers’ reports

• Recipient

Report Status did not update correctly from “Certified” to “Submitted” status

9

AJ There were also some challenges that occurred during the submission of the Recipient Report. As I’m sure you have seen in some of our previous presentations, this is the first year that agencies could begin submitting client-level data for additional services provided using RWHAP-related funding (including program income

• r pharmaceutical rebates) under what’s called Eligible Services Reporting. Many

recipients began including information on these services in their contracts but with this change there were a number of recipients that needed some assistance or clarification on what the new reporting requirement is and what they need to do in

• rder to be reporting correctly. As a reminder, Eligible Services Reporting is not

required until the 2021 RSR (submitted in March 2022) and while we won’t go into any further detail today if you would like some more information on the topic I highly recommend checking out the RSR in Focus document on the TargetHIV website linked

• n this slide.

Some recipients experienced an unexpected issue when attempting to access their RSR inbox from the Electronic Handbooks. These recipients were being redirected to an incorrect inbox for either another agency’s grant or a different grant of their own 9

than they had selected. This prevented these recipients from being able to access their own and their subrecipients’ reports but thankfully the issue was easily corrected in each case by logging out and back into the system. Additionally, towards the end of the submission period, a few recipients encountered an issue with the status of their Recipient Report not updating correctly. Once all Provider Reports are accepted by a recipient, the Recipient Report should automatically move from ‘Certified’ to ‘Submitted’ status. However, these Recipient Reports did not automatically move to ‘Submitted’ status. We worked with the system developers to get the issue resolved and have these reports advanced to the correct status. 9

EHB/Web System Recipient Report Provider Report Client-level Data

Provider Report Multiple Zip Code Files

• Providers

with multiple data files for the new Clients by ZIP Code section were

• nly

able to upload a single file to the system

• When

an additional file is uploaded, the previous file’s data were

• verwritten

10

Let’s move on to the Provider Report. This year there was a new section in the Provider Report, Clients by ZIP Code. Being a new section, many agencies ran into some challenges trying to upload their data correctly. For this section, providers can upload a .csv file using the provided template which includes their ZIP Code. Some agencies though encountered an issue when attempting to upload multiple files. The system only accepted one file at a time and therefore when an additional file was uploaded, the previous one’s data would be erased and overwritten. The DART Team worked with multiple agencies to merge multiple ZIP Code data files and when that wasn’t possible, upload the largest of the multiple files that each agency had. 10

EHB/Web System Recipient Report Provider Report Client-level Data

Provider Report ZIP Code Template

• Some

providers experienced additional problems when attempting to upload their ZIP Code files with:

• Incorrect

capitalization in .csv file column headers

• A

single ZIP Code listed more than

• nce
• ZIP

Codes with leading zeroes

11

We also identified a number of other smaller issues that some organizations ran into when trying to upload their ZIP Code data. In each case, the file would fail to upload,

• r the data populated in the system would be incorrect. These occurred when files

had incorrect capitalization in the column headings that differed from the supplied template, listed a single ZIP Code more than once in the file, or had ZIP Codes with leading zeroes. We worked to format the affected files for each agency so that they were able to successfully upload. 11

EHB/Web System Recipient Report Provider Report Client-level Data

Provider Report Additional Services Table

• Additional

Services table added to the Service Information section

• Providers

should select any additional services funded with their

• wn

RWHAP-related funding (program income

• r

pharmaceutical rebates)

12

To align with Eligible Services Reporting, there was also a new table added to the Service Information section of the Provider Report. The Additional Services table is located at the bottom of the Service Information page. Using this table, providers can select any additional services they provide using their own RWHAP-related funding (including program income or pharmaceutical rebates). We noticed several providers who were using this table incorrectly and selected any services that their agency provided regardless of funding source. As a result, these providers would receive a validation message for each service they checked off that they did not upload client-level data for. As a reminder, providers should only use this table to select services provided by their own RWHAP-related funding. 12

EHB/Web System Recipient Report Provider Report Client-level Data

Provider Report DUNS Number

• New

warning validation message for missing DUNS number in Provider Report

13

One last point to make about the Provider Report. This year there was a new validation in the system where providers would receive a warning if their report did not have a DUNS number listed. We noticed that many providers did not know where this information could be updated. This field is listed in the very first section of the report, General Information, and to add or edit the value listed, simply click the **“Update” link as shown on the screenshot here to make any necessary changes. 13

• EHB/Web System

Recipient Report Provider Report Client level Data

Client-Level Data New System Schema

• There

were changes to the RSR schema this year, which inevitably cause some issues

Check if your XML’s data elements are current Check the schema version number – this year we updated to version 5.1

14

Finally, we wanted to go over some of the issues related to the client-level data upload. There were several changes to the 2019 RSR, including new data elements and revisions to some existing elements. As a result, the RSR schema were updated to reflect the changes and a new version of the schema was released. This meant that all providers had to update their data systems to create a compliant file. What you’re seeing on this screen is a schema error for a file that was created using an outdated version of CAREWare. You can check **the schema number to make sure it lines up with the most up-to-date version, and you can also look to see if the **data elements themselves look correct. 14

• EHB/Web System

Recipient Report Provider Report Client level Data

Client-Level Data “Processed with Errors”

• Client-level

data files were rejected for mismatched OrgID/RegCode

Clients are not added to the report if the file is “Processed with Errors”

15

AJ The most common issue that we saw with the client-level data file upload this year was related to a new schema check that was added to the RSR web system that checked the Provider Report’s OrgID and RegCode against those same fields in the client-level data file. Because these fields are optional in the client-level data file, many agencies do not include them (or, for CAREWare users, a placeholder is added if the values are not entered manually into CAREWare), meaning that the system rejected many files early on the submission season. We worked with the system team to remove the schema check quickly after the system was opened. 15

• EHB/Web System

Recipient Report Provider Report Client level Data

Client-Level Data Systems Creating the RSR XML

• TRAX

crashed in some instances

• Some

RSR-Ready Systems struggled to implement eligible services reporting

• Make

sure you are using the most up-to-date build

• f

your data system

CAREWare 6

• Updates

due to schema changes

16

AJ Finally, we also had a few problems with data systems that create the RSR client-level data file. TRAX, which is a free application that created the RSR client-level data file from structured CSV data, crashed in some instances early in the reporting period. Some systems also had issues in updating to meet the new eligible services reporting requirements. Make sure that you are using the most recent version of your data system (or TRAX) – this was a challenges this year for many CAREWare users who had to migrate to CAREWare 6. 16

Poll Question #2

17

What type of challenges did your agency encounter during the 2019 RSR reporting cycle? Please select all that apply.

• Missing data
• Difficulty understanding validation messages
• Communication issues between recipients and providers
• Questions or concerns about the submission process
• Technical challenges RSR web system, e.g uploading CLD

Thanks Ruchi. [contextualize responses] Let’s move on and talk about some of the submission challenges we identified during this reporting season. AJ passes to Brian 17

Webcast Overview (3)

2019 RSR Submission Statistics RSR Submission Challenges RSR Comment Review Next Steps and Resources

18

Brian Next, we’re going to turn to RSR comment review updates from the 2019 RSR. We’ll turn things back to Jennifer to explain challenges related to system validations. 18

2019 RSR Comment Review

19

JENNIFER MENTION HOUSING STATUS COLLECTED DATE Every year Data Support reviews the numerous comments you all entered into the system as part of your RSR system. This year Data Support has reviewed 15,916 comments including responses to warning validations as well as any feedback you all have entered as part of the submission step. The graphic on this slide shows the most frequent validation message categories that we saw in the system. 40% of these validations were related to services data (which we’ll review on the upcoming slides). 25% of these messages were for missing clinical data such as CD4 or viral load test results and 22% were for missing demographic data such as poverty level or housing status. Also of note on this graph is that 7% of these messages were for missing housing status collected date. This was a new data element for this year and based on our review a number of organizations really struggled to collect and/or report this data element correctly. As part of the comment review process, RWHAP technical assistance teams will be contacting agencies in the coming weeks to provide assistance to those that indicated significant problems, need additional guidance on reporting requirements, and/or had problems with their data completeness. 19

Thanks to everyone for the feedback that you all provide as part of your RSR

• submission. These comments help us as TA providers to better identify your needs

and where we can make improvements for future reporting periods. 19

Services Validation Messages

• Many

providers still struggle to understand these three related validation messages

• Service(s)

delivered but not uploaded

• Service(s)

funded but not delivered

• Services(s)

uploaded but not delivered

20

So as a result of the comment review process, I do want to highlight a few validation messages we frequently see that many recipients and providers still struggle to

• understand. These validation messages are all regarding services data and include

services delivered but not uploaded, services funded but not delivered, and services uploaded but not delivered. Let’s take a look. 20

• Reci ient listed th

Checked off as p e Uploaded as part service as funded delivered in

• f client level

Provider Report data Service(s) funded but not delivered Service(s) delivered but not uploaded Service(s) uploaded but not delivered

Service Validation Messages Explained

21

By taking a look at this table hopefully these messages will make a bit more sense to all of you out there that are still struggling to understand them. We’ll start with services funded but not delivered. This validation message indicates that your recipient(s) have indicated the service category as being funded in their contracts, but the provider did not select the service as being delivered in the Service Information section of their report. Services delivered but not uploaded essentially indicates missing data. The service category has been checked off as being delivered in the Service Information section

• f the Provider Report, but the client-level data do not include any data on the

indicated service category. Services uploaded but not delivered is the opposite of that. This validation indicates that your client-level data file includes data on the indicated service category, but it is not checked off as delivered in the Service Information section of the Provider Report. 21

Recipient-Provider Coordination

• Recipients

and providers can frequently coordinate together to resolve these validation messages

• If

a service is errantly listed as funded

• r

missing in your Provider Report, work with your recipient so they can add

• r

remove the services as needed

• Contact

Data Support if you need addit ional assistance altering the services in your contracts/report

22

So what can be done about these validations. Sometimes a validation comment is an appropriate response, but oftentimes coordination between recipients and providers is the best way to resolve these validation messages. Services may be missing or listed incorrectly in an agency’s Provider Report. In either case, providers should let their recipient know and work with them to ensure the services listed are as accurate as possible. We want your data to accurately reflect the wonderful work that you all do so if your report is missing data for a service that you all don’t deliver, reach out to your recipient to see if it should be removed from your report. And if you have data for services you’re providing using RWHAP funding but the service category isn’t listed for you to check off as delivered, talk to your recipient to see if it should be added to their contracts. Additionally, if you need assistance with these validation messages or how to proceed if or when your receive one, please don’t hesitate to reach out to Data Support using the contact information we’ll display at the end of presentation and we can help you figure out the best course of action. 22

So before we move on to the rest of the presentation, let’s go to one last poll

• question. Ruchi?

22

Poll Question #3

23

Did your agency experience validation issues during the process of submitting your 2019 RSR?

• A. No, process was smooth; submitted on-time with no issues
• B. Yes, a few issues, but able to resolve before submitting
• C. Yes,

and we were unable to correct data before submitting Thanks Ruchi. [contextualize responses]. Brian passes to Ellie 23

Webcast Overview (4)

2019 RSR Submission Statistics RSR Submission Challenges RSR Comment Review Updates Next Steps and Resources

24

Ellie To wrap up the presentation, I’ll turn things over to Ellie to go over the TA Resources as well as some other helpful information for you all. 24

Reminders for RWHAP Recipients & Subrecipients

• COVID-19

Data Report is required for all recipients and subrecipients

• f

CARES Act funding

• CDR

Resources

• n

TargetHIV

• Eligible

Services Reporting will be required for the 2021 RSR, submitted in March 2022

• In

Focus Document

• n

Eligible Services Reporting

• Understanding

Reporting Changes webinar (Slides 6-9)

25

ELLIE Thanks Brian. First, as many of you may know HAB is now collecting data from all recipients and subrecipients of CARES Act funding through the COVID-19 Data Report,

• r CDR. Data Support has recently developed an Instruction Manual for this report,

which you can find along with a number of other really helpful resources on TargetHIV. Also, we want to remind you that Eligible Services reporting will be required for the 2021 RSR (submitted in March 2021). We will continue to discuss this in our Fall webinar series, but you can get a head start understanding this important new requirement with the resources included on this slide. 25

DART will soon be DISQ!

• On

July 1st

the

• rganizations

that make up the DART Team will begin a new cooperative agreement as the Data Integration, Systems, and Quality (DISQ) Team

• You

can still reach us at data.ta@caiglobal.org

• Our

expanded scope will include supporting RWHAP recipients and subrecipients with the CDR and Ending the Epidemic (EHE) triannual module

26

ELLIE Lastly, we want to let you all know that CAI, Mission Analytics, Abt Associates and Debbie Isenberg – the folks who make up the DART Team – have been funded to expand our scope beginning this July. As of July 1st, we will be the DISQ Team – which stands for Data Integration, Systems, and Quality Technical Assistance. You can still reach us at our regular email address, but you will be able to get guidance from us on multiple new reports, including the CDR and the Ending the Epidemic triannual modules, and we’ll have a special emphasis on helping you all improve your data quality and use your data in ways that helps your organizations thrive. We hope you’ll continue reaching out! 26

27

TA Resources

TA Resource Type of TA

Ryan White Data Support

• RSR-related

content and submission questions 888-640-9356

• Interpretation
• f

the RSR Instruction Manual and HAB’s reporting requirements

RyanWhiteDataSupport@wrma.com

• Instructions

for completing the RSR Recipient and Provider Reports

• Data

validation questions DART/DISQ

• Data

reporting requirements Data.TA@caiglobal.org Creating the RSR client-level data file

• TRAX,

CHEX and the encrypted Unique Client Identifier Application

• Data

quality issues

• EHBs Customer Support Center
• RSR

software-related questions 877-464-4772

• Electronic

Handbook (EHB) navigation http://www.hrsa.gov/about/cont • EHB registration act/ehbhelp.aspx

• EHB access and permissions
• Performance Report submission statuses
• RSR Web System navigation

CAREWare Help Desk

• How

to generate the XML file from CAREWare correctly 877-294-3571

• How

to view a sample client summary file cwhelp@jprog.com

• Creating

custom reports

RSR Data TA Brochure: https://targethiv.org/library/rsr-data-ta-brochure

ELLIE The Data Support team addresses RSR-related content and submission questions, including interpretation of the RSR Instruction Manual and HAB’s reporting requirements. The DART team – soon to be the DISQ Team – addresses questions for those needing assistance in extracting data to meet data reporting requirements, assisting recipients in extracting data from their systems, and helping with data quality issues. The EHBS Customer Support Center assists recipients with software-related questions, such as registering for and navigating the EHBs, resetting passwords, and making sure that you have the right permissions to complete the reports. For CAREWare users, the CAREWare Help desk is the best resource for all CAREWare related questions. If you are unsure of who to call, feel free to contact any one of the resources provided and they will be able to direct you to the appropriate place. 27

Connect with HRSA

To learn more about

• ur

agency, visit www.HRSA.gov

Sign up for the HRSA eNews

FOLLOW US:

28

ELLIE Finally, to connect with and find out more about HRSA, check out HRSA.gov. We are now going to move on to the Q&A portion of the webinar. We invite you to join in on the discussion during the Q&A and let us know about your own RSR submission experience.

• What problems did you have?
• What suggestions do you have to make the web system or the submission process

run smoother? We would love to hear from you. Thank you to everyone for joining us today. 28

Questions

• r

Comments?

• Please

use the “raise hand” function to speak. We will unmute you in the

• rder

that you appear. OR

• Type

your question

• r

comment in the question box.

29

ELLIE I want to give a final thanks to AJ and Jennifer for their great presentation. We’ve now come to the question and answer portion of the webinar. As a reminder, you can send us questions using the “Question” function on your control panel on the right-hand side of the screen. You can also ask questions directly “live.” You can do this by clicking the “raise hand” button (on your control panel). If you are using a headset with a microphone, my colleague Beth will conference you in; or, you can click the telephone button and you will see a dial-in number and code. We hope you consider asking questions “live,” because we really like hearing voices other than our own. One other reminder before we start reviewing the questions. In addition to answering questions live, we also write written answers that are posted with the slides on the TargetHIV website within two weeks of the webinar. Remember that the written answers always trump the verbal ones that we provide live, so be sure to check the information that is posted. We have a brief evaluation that will appear as you exit the webinar. We review your feedback after every webinar to better understand what you liked, any areas for 29

improvement as well as additional information you would like included in the future. We appreciate you takin the time to complete this. 29