Statistical Sampling and Extrapolation: Challenging Methods and - - PowerPoint PPT Presentation

Medicare and Medicaid Audits Using Statistical Sampling and Extrapolation: Challenging Methods and Results

Today’s faculty features:

1pm Eastern | 12pm Central | 11am Mountain | 10am Pacific

The audio portion of the conference may be accessed via the telephone or by using your computer's

• speakers. Please refer to the instructions emailed to registrants for additional information. If you

have any questions, please contact Customer Service at 1-800-926-7926 ext. 1.

THURSDAY, JUNE 14, 2018

Presenting a live 90-minute webinar with interactive Q&A Anna M. Grizzle, Member, Bass Berry & Sims, Nashville, Tenn.

• Dr. Patricia L. Maykuth, Ph.D, President, Research Design Associates, Decatur, Ga.

Tips for Optimal Quality

Sound Quality If you are listening via your computer speakers, please note that the quality

• f your sound will vary depending on the speed and quality of your internet

connection. If the sound quality is not satisfactory, you may listen via the phone: dial 1-866-927-5568 and enter your PIN when prompted. Otherwise, please send us a chat or e-mail sound@straffordpub.com immediately so we can address the problem. If you dialed in and have any difficulties during the call, press *0 for assistance. Viewing Quality To maximize your screen, press the F11 key on your keyboard. To exit full screen, press the F11 key again.

FOR LIVE EVENT ONLY

Continuing Education Credits

In order for us to process your continuing education credit, you must confirm your participation in this webinar by completing and submitting the Attendance Affirmation/Evaluation after the webinar. A link to the Attendance Affirmation/Evaluation will be in the thank you email that you will receive immediately following the program. For additional information about continuing education, call us at 1-800-926-7926

• ext. 2.

FOR LIVE EVENT ONLY

Program Materials

If you have not printed the conference materials for this program, please complete the following steps:

• Click on the ^ symbol next to “Conference Materials” in the middle of the left-

hand column on your screen.

• Click on the tab labeled “Handouts” that appears, and there you will see a

PDF of the slides for today's program.

• Double click on the PDF and a separate page will open.
• Print the slides by clicking on the printer icon.

FOR LIVE EVENT ONLY

Medicare and Medicaid Audits Using Statistical Sampling and Extrapolation

Anna Grizzle - Bass, Berry & Sims, PLC Pat Maykuth, Ph.D. - Research Design Associates, Inc.

Use of Statistical Sampling for Overpayment Estimation

• Medicare or Medicaid audit

– MAC audits following TPE reviews – ZPIC or UPIC audits – OIG audits – Medicaid agency audits – MIC audits

• OIG self-disclosure protocol
• Internal compliance audit
• Calculation of damages in FCA case?

6

Legal Basis for Statistical Sampling for Overpayment Estimation

“The use of statistical sampling to project an

• verpayment. . . does not deny a provider or

supplier due process. Neither the statute nor regulations require that a case-by-case review be conducted in order to determine that a provider or supplier has been overpaid and to determine the amount of overpayment.” HCFA Ruling 86-1

7

Legal Basis for Statistical Sampling for Overpayment Estimation

Statistical sampling does not violate due process “so long as extrapolation is made from a representative sample and is statistically significant.” Chaves County Home Health Service, Inc. v. Sullivan, 931 F.2d 914 (D.C. Cir. 1991), cert. denied, 402 U.S. 1091 (1992).

8

Legal Basis for Medicare Statistical Sampling and Extrapolation

A Medicare contractor may not use extrapolation to determine overpayment amounts . . . unless . . . – There is a sustained or high level of payment error; or – Documented educational intervention has failed to correct the payment error 42 U.S.C. §1395ddd(f)(3)

9

Legal Basis for Medicare Statistical Sampling and Extrapolation

• The PIM provides basic concepts rather than

a checklist to complete a valid statistical sampling.

– When applied correctly, the PIM’s concepts can lead to a proper methodology to use as a basis for extrapolation.

• The PIM’s concepts often are not applied

correctly in developing the statistical sampling methodology used as the basis of extrapolation in Medicare audits.

Source: Chapter 8 – Benefit Integrity; Medicare Program Integrity Manual; available at: http://www.cms.gov/manuals/downloads/pim83c08.pdf 10

Legal Basis for Medicaid Statistical Sampling and Extrapolation

• Dictated by state law
• If no explicit authority, look to due process

requirements

11

Performance of Statistical Sampling and Extrapolated Overpayment

• Major Steps

– Selecting the provider or supplier

• Prior history of overpayment error
• TPE Review by MACs

– Selecting the period to be reviewed – Defining the universe, the sampling unit, and the sampling frame

• Define the provider, issue to be reviewed, time

period, and methodology for measuring

• verpayment

Source: Chapter 8 – Benefit Integrity; Medicare Program Integrity Manual; available at: http://www.cms.gov/manuals/downloads/pim83c08.pdf 12

Performance of Statistical Sampling and Extrapolated Overpayment

• Major Steps (cont.)

– Designing the sampling plan and selecting the sample – Reviewing each of the sampling units and determining if there was an overpayment

• r under payment

– Estimating the overpayment

Source: Chapter 8 – Benefit Integrity; Medicare Program Integrity Manual; available at: http://www.cms.gov/manuals/downloads/pim83c08.pdf 13

Documentation Requirements

• The following items must be documented:

– Universe – Sampling frame (sorted) – Random sampling process (seed, program, inputs and printout) – Sample size determination calculations – Extrapolation formulae, inputs and printouts – Extrapolation recalculation where appropriate

14

Valid Methodologies

• MPIM specifically lists:

– Simple random – Stratified random – Cluster

• MPIM specifically requires proper execution of

chosen methodology

15

Valid Statistics

• Statistically Valid Random Sample (SVRS)
• Probability Sample
• Use correct formulae
• Follow statistical requirements of chosen

statistics

• Point Estimates and Confidence Intervals are

valid statistics – if properly executed

• Minimum Sum Method and Penny sampling not

been validated in the statistical literature

16

Probability Statistics

• To know the number of samples of the chosen size

that can be created from the frame

• Known likelihood of selection of each sampling unit
• Proper randomization
• Proper execution of sample methodology
• Use correct formulae
• Accurate measurement of the variable of interest

(overpayment)

Source: Chapter 8 –§ 8.4.2 Benefit Integrity; Medicare Program Integrity Manual; available at: http://www.cms.gov/manuals/downloads/pim83c08.pdf 17

Frame x Sample (1, 2, 3, 4, 5) (2, 3, 4, 5, 6) (1, 11, 21, 31, 41) (1, 12, 23, 34, 45) (8, 23, 46, 73, 90) (1, 2, 3, 4, X) Outside of frame 18

Probability Sample

• 1. To be able to identify the number of

samples of a given size that can possibly be selected from a frame of a frame of a given size claims. How many samples of 5 can be selected from a frame of 100?

• 2. In a simple sample each claim must have a

known and equal probability of selection. Strata take simple samples from strata frames.

19

Probability Yardstick

• Single sample chosen for the audit is only one sample (of the chosen

size) out of a large number of possible samples

• Sampling distribution of all possible means provides mathematical

model of what is likely to occur if all possible samples analyzed

• If a large number of samples were drawn from the frame:

– A mean can be calculated for each sample – Each sample mean would not be exactly the same value as others – Means would be different from one another but would cluster around the frame’s central value (or mean) – Differences in the means of different samples are basis of the error that occurs inferring from a sample to the frame rather than measuring all of the claims in the frame – If repeated random samples (moving toward infinity) were made, means would be expected to fall into a normal distribution

20

Possible Samples That Can Be Randomly Drawn From the Frame

21

What’s the Big Deal About the Normal Distribution?

• In probability theory, the normal (or bell-shaped) distribution

is continuous probability distribution (a function that tells the probability of a number falling between any two real numbers). The normal distribution is symmetric around the

• mean. The mean, median and mode are the same number.
• The normal distribution is immensely useful because of the

Central Limit Theorem which states that the mean of many random variables independently drawn from the same distribution is distributed approximately normally, irrespective of the form of the original distribution. That is, the overpayment means will be randomly distributed if the sample is large ... moving toward infinity.

22

“Normal” Distribution

23

Confidence Levels

• n Normal Distribution

Lower Confidence Limit Point Estimate Upper Confidence Limit

24

25

Mode Mean Median 12,072.62 566.09 1,274.35

Where is the Confidence Level? One-sided or Two

Illustration not based on actual data

26

Distribution of Frame

Paid Dollars

27

Non Normal Mean and Standard Deviation

Sample mean +/- 1 standard deviation Mean = 430.92 + sd = 525.51 = 956.43 430.92 - 25.51 = -94.18

28

Confidence Levels Non Normal Data

Point Estimate +/- ½ Confidence Interval Confidence Interval = 16,778 + 16,778 = 33,556 Lower 27,147 + 16,778 = $43, 925 Upper 27,147 - 16,778 = $ 10,368

Lower confidence Level ??? Unknowable

29

Properly Executed Random Sample

• Prior known error rate or probe
• Appropriate for use with the audit data

(dependent or independent)

• Of sufficient size to support the statistic used
• Use a replicatable random process
• Tested for randomness
• Is representative (without mathematical bias) and

tested

30

31

Sample Size Determination Based on Chosen Precision and Confidence

32

RAT-STATS Results

80% 90% 95% 99% 1% 75 77 78 79 2% 63 68 71 75 5% 29* 39 46 56 10% 10* 15* 20* 29* 15% 5* 8* 10* 16*

Confidence Level Precision Level

* Caution sample sizes less than 30

33

Representative Samples

• A small yet complete and accurate picture of the

data in the frame.

• A subset of a statistical universe that accurately

reflects the numerical membership of the entire universe and its distribution.

• A representative sample is an unbiased indication of

what the frame is like.

• Representativeness is tested mathematically. When

a sample is not representative, the result is known as a sampling error.

34

Representative Sample

Frame sample

35

Sobering How Poor a Sample Can Be

36

RAT-STATS Overpayment Estimation

• Formulae:
• Confidence Level:

37

Precision vs Accuracy

• Precision measures how often the measurement tool

produces a similar result every time it is used.

• Accuracy measures how close the sample value

comes to the true value. How close is the sample

• verpayment mean to the true mean of
• verpayments in the frame.
• Sample results can be accurate but not precise or

precise but not accurate

• Extrapolations require that the results meet a

predetermined level of both.

38

Precision Calculation

Point Estimate +/- Precision amount = Confidence Level Precision amount/Point estimate = Precision % It is absurd to say the lower confidence level benefits the

• provider. The accuracy of the estimate is only the top and

bottom of the confidence level around the point estimate. The lower confidence level is as close to accurate

• verpayment that can be obtained.

39

Sample of 5

If:

• A random sample of 5 selects the top most expensive

claims (5540, 5550, 5555, 5560, 5570) mean = 5555

• Frame of 100 ranges from 500 to 5570 has a mean = 2827
• The precision will very small and the confidence levels

will be very tight but the point estimate would be nearly twice as large as the frame mean

• The precision would be good but the accuracy would not
• Point estimates can exceed the amount actually paid
• It is a bad estimate.

40

41

Sample of 5

If:

• A random sample of 5 selects the top 2 claims and bottom 3

(500, 525 540, 5560, 5570) mean = 2539

• Frame of 100 ranges from 500 to 5570 has a mean = 2827
• The precision will very large and the confidence levels will be

very large but the point estimate would be close to the frame mean

• The precision would be unacceptable but the accuracy would

be OK

• The lower confidence level can be a negative number

indicating an underpayment

• It is a bad estimate

42

Poor Audit Design & Execution Produce Only “Invalid Results”

Statistics in the hands of an inept auditor are like a lamppost to a drunk: they are used more for support than illumination.

43

Defending Against Extrapolation Results

• Medicare Appeals Process

– Redetermination – Reconsideration – Administrative Law Judge Hearing – Medicare Appeals Council – Federal District Court

• Medicaid Appeals Process

– Appeal rights under state law

44

Defending Against Extrapolation Results

• No administrative or judicial review of

determination of high level of payment error BUT determination must be made

• Failure to follow one or more requirements in

Benefit Integrity Manual does not necessarily affect validity

• Not sufficient to argue better or more precise

methods are available

45

Defending Against Extrapolation Results

• Can challenge validity of sampling

methodology based on “the actual statistical validity of the sample as drawn and conducted”

• Test: Was the sample statistically valid?
• Contractor has burden of establishing sample

was in fact random and statistically valid

46

Defending Against Extrapolation Results

• Procedural Challenges

– Did the contractor provide work papers allowing for review and replication of sampling results for every stage of the process? – Did the contractor follow the guidelines?

• Medicare: MPIM
• Medicaid: State requirements

– Were allowed claims included in overpayment sample calculation? – Were calculations performed correctly in the audit and at each level of appeal?

47

Defending Against Extrapolation Results

• Substantive Challenges

– Need a statistician to make your case.

• Where can you find one?

– “One size does NOT fit all.” – It is not your job to explain how it should be done.

48

Defending Against Extrapolation Results

• Examples of Substantive Challenges

– Is the sample representative? – Is the sample statistically significant?

• Is the sample size reliable?
• Is the sample within the required

precision and confidence levels?

49

Hold Auditors Accountable

• Did they know what should be done statistically?

– Requires consultation with statistical expert

• Did they do what is necessary to create and audit a

probability sample (not just say they did)?

• Have to test the data to be sure
• Did they accurately follow chosen methodology key

assumptions (accurately execute the methodology, use proper randomization, use the correct formulae and accurately measure variables of interest)?

• All of these criteria must be met for extrapolation
• Did they select a random sample of sufficient size made up
• f independent observations that are normally

distributed, randomly selected and representative?

50

Common Substantive Issues

• Sample size not associated with established precision
• r confidence levels
• Incorrect use of formulas
• Use of wrong formulas - choose wrong method
• Use of inapplicable methodology – simple, stratified,

cluster, multi-stage

• Non-representative sample
• Fail to meet key assumptions of statistics – math basis
• f statistics
• Exclusion of zero paid claims
• Accuracy outside of recommended range – too little

precision

• Reporting incorrect precision and/or confidence levels

51

Defending Against Extrapolation Results

• Obtain all documentation related to sampling calculations

– Consider provider’s prior audit history

• Know appeal timelines and requirements for each level

– No new evidence after reconsideration absent good cause – Request for ALJ Hearing – 42 C.F.R. § 405.1014(a)(3)

• Include information on each sample claim to be appealed
• File request within 60 calendar days after the party

receives the last reconsideration for the sample claims

• Assert the reasons for disagreement with how the

statistical sample and/or extrapolation was conducted in the request for hearing

52

Defending Against Extrapolation Results

• Understand reasons for denial at each level
• Present reasons in written protest or position

paper

• Prepare for oral testimony at ALJ hearing

53

Questions?

54

Faculty

Anna M. Grizzle, Esq. Member Bass, Berry & Sims PLC 150 Third Avenue South Suite 2800 Nashville, TN 37201 agrizzle@bassberry.com (615) 742 7732 Pat Maykuth, Ph.D. President Research Design Associates, Inc. 721 E Ponce de Leon Decatur, GA 30030 www.researchdesignassociates.com pm@researchdesignassociates.com (404) 373 4637

55