Operating characteristics of frequently used similarity rules - - PowerPoint PPT Presentation

Florian Klinglmueller* Ack: Andreas Brandt, Thomas Lang, Ina Rondak

Operating characteristics of frequently used similarity rules

*Austrian Medicines & Medical Devices Agency

The contents of this presentation are my personal opinion. My remarks do not necessarily reflect the official view of AGES.

• Comparison of an originator product to a biosimilar product with respect to

„critical quality attributes“ (i.e. physical, chemical, biological, or microbiological properties that ensure product quality) with the aim to conclude similarity on the quality level.

• One quality attribute on a continuous scale. Comparison between samples from
• riginator and biosimilar.
• Different rules to decide whether samples from the biosimilar are similar to

samples from the originator - based on sample data

• Explore operating characteristics of commonly used rules under different scenarios
• f similarity and dissimilarity.

Similarity assessment of quality attributes

Introduction

Decision rule

• Problem: Similar is not equal!
• ⇒ Specify what is similar enough
• Average similarity
• E.g.: Biosim on average within ± 1.5

standard deviations of reference mean

• E.g.: Ratio of CQA on average 80%-125%
• Population based similarity
• Reference samples define margin of what

is safe (e.g. min-max, TI)

• Future/observed biosim batches fall into

range

• Translate into a criterion

How to compute „similar/not similar“ from data

Decision rule

• Problem: Similar is not equal!
• ⇒ Specify what is similar enough
• Average similarity
• E.g.: Biosim on average within ± 1.5

standard deviations of reference mean

• E.g.: Ratio of CQA on average 80%-125%
• Population based similarity
• Reference samples define margin of what

is safe (e.g. min-max, TI)

• Future/observed biosim batches fall into

range

• Translate into a criterion

How to compute „similar/not similar“ from data

Decision rule

• Problem: Similar is not equal!
• ⇒ Specify what is similar enough
• Average similarity
• E.g.: Biosim on average within ± 1.5

standard deviations of reference mean

• E.g.: Ratio of CQA on average 80%-125%
• Population based similarity
• Reference samples define margin of what

is safe (e.g. min-max, TI)

• Future/observed biosim batches fall into

range

• Translate into a criterion

How to compute „similar/not similar“ from data

• Min-Max: gives a the range of the observed values. Purely descriptive i.e. permits

little inference about future samples of the process, except that true range is wider

• X-SD: estimates the variation in the sample around the sample mean. Purely

descriptive, many statistical intervals are constructed by choosing x such that probabilistic statements hold

• E.g. 95% CI: x=1.96; 95% PI (n=10): x=2.16, 95/95 TI (n=10): x=3.38
• Confidence Interval: estimates a range that should cover an unknown parameter

(e.g. mean) of the distribution assumed to generate the data

• Prediction Interval: estimates a range that should cover the value of the next

sample from distribution assumed to generate the data

• β-content Tolerance Interval: estimates a range that should cover a certain

proportion β of future samples from the distribution assumed to generate the data

Statistical Intervals

Probabilistic interpretation of different interval types

• Min-Max: gives a the range of the observed values. Purely descriptive i.e. permits

little inference about future samples of the process, except that true range is wider

• X-SD: estimates the variation in the sample around the sample mean. Purely

descriptive, many statistical intervals are constructed by choosing x such that probabilistic statements hold

• E.g. 95% CI: x=1.96; 95% PI (n=10): x=2.16, 95/95 TI (n=10): x=3.38
• Confidence Interval: estimates a range that should cover an unknown parameter

(e.g. mean) of the distribution assumed to generate the data

• Prediction Interval: estimates a range that should cover the value of the next

sample from distribution assumed to generate the data

• β-content Tolerance Interval: estimates a range that should cover a certain

proportion β of future samples from the distribution assumed to generate the data

Statistical Intervals

Probabilistic interpretation of different interval types

Frequentist confidence: in repeat experimentation range estimate computed in this way will cover the quantity (parameter, next sample, all future samples) a certain proportion of times (e.g. 95%)

Rules for concluding biosimilarity

• Min-Max: All samples of the biosim are between min-max of the originator
• X-Sigma: All samples from the biosim are within x-standard deviations of the
• riginators mean
• (75%/90%) Tolerance interval: All samples from the biosim are within a P/Q

Tolerance interval of the originator

• TI Specs: The P/Q tolerance interval of the biosim is within „specifications“ (e.g.

Min-Max) of the originator

• FDA Rule: The 90% confidence interval for mean difference between originator

and biosim is within a similarity margin of 1.5 standard deviations of originator A selection of frequently used decision rules

Differences in mean, equal variance

Simulation scenario 1

• Originator and Biosim samples follow

standard normal distribution

• Equal variance
• Distance between distributions

expressed as multiples of the (common) standard deviation

• Settings considered:
• M2-M1= {0, 0.5, 0.8, 1, 1.5} * SD

Differences in mean, equal variance

Simulation scenario 1

• Originator and Biosim samples follow

standard normal distribution

• Equal variance
• Distance between distributions

expressed as multiples of the (common) standard deviation

• Settings considered:
• M2-M1= {0, 0.5, 0.8, 1, 1.5} * SD

Differences in mean, equal variance

Simulation scenario 1

• Originator and Biosim samples follow

standard normal distribution

• Equal variance
• Distance between distributions

expressed as multiples of the (common) standard deviation

• Settings considered:
• M2-M1= {0, 0.5, 0.8, 1, 1.5} * SD

Differences in mean, equal variance

Simulation scenario 1

• Originator and Biosim samples follow

standard normal distribution

• Equal variance
• Distance between distributions

expressed as multiples of the (common) standard deviation

• Settings considered:
• M2-M1= {0, 0.5, 0.8, 1, 1.5} * SD

• Equal means (biosimilar) shown with dashed lines, unequal means solid lines.
• Probability to conclude similarity decreases with increasing dissimilarity (m=10,n=10)

Simulation results: Equal variances

Most simple scenario

• Equal means (biosimilar) shown with dashed lines, unequal means solid lines.
• Probability to conclude similarity decreases with increasing dissimilarity (m=10,n=10)
• If we increase the sample size (m=20,n=20) several things happen

Simulation results: Equal variances

Most simple scenario

• Equal means (biosimilar) shown with dashed lines, unequal means solid lines.
• Probability to conclude similarity decreases with increasing dissimilarity (m=10,n=10)
• If we increase the sample size (m=20,n=20) several things happen
• Similarity conclusions increase, with increasing originator samples (except TI)

Simulation results: Equal variances

Most simple scenario

• Equal means (biosimilar) shown with dashed lines, unequal means solid lines.
• Probability to conclude similarity decreases with increasing dissimilarity (m=10,n=10)
• If we increase the sample size (m=20,n=20) several things happen
• Similarity conclusions increase, with increasing originator samples (except TI)
• With increasing Biosim samples similarity conclusions increase for for interval based

criteria

Simulation results: Equal variances

Most simple scenario

Difference in means, unequal variance

Simulation scenario 2

• Originator and biosim distribution may

differ in mean and in variance

• Standard deviation of originator is

sdratio times larger than biosim

• Values for sdratio: .25 - 2
• sdratio=1 corresponds to

Scenario 1

• Both cases, assuming equal and

unequal means, were investigated

Difference in means, unequal variance

Simulation scenario 2

• Originator and biosim distribution may

differ in mean and in variance

• Standard deviation of originator is

sdratio times larger than biosim

• Values for sdratio: .25 - 2
• sdratio=1 corresponds to

Scenario 1

• Both cases, assuming equal and

unequal means, were investigated

Difference in means, unequal variance

Simulation scenario 2

• Originator and biosim distribution may

differ in mean and in variance

• Standard deviation of originator is

sdratio times larger than biosim

• Values for sdratio: .25 - 2
• sdratio=1 corresponds to

Scenario 1

• Both cases, assuming equal and

unequal means, were investigated

Difference in means, unequal variance

Simulation scenario 2

• Originator and biosim distribution may

differ in mean and in variance

• Standard deviation of originator is

sdratio times larger than biosim

• Values for sdratio: .25 - 2
• sdratio=1 corresponds to

Scenario 1

• Both cases, assuming equal and

unequal means, were investigated

Difference in means, unequal variance

Simulation scenario 2

• Originator and biosim distribution may

differ in mean and in variance

• Standard deviation of originator is

sdratio times larger than biosim

• Values for sdratio: .25 - 2
• sdratio=1 corresponds to

Scenario 1

• Both cases, assuming equal and

unequal means, were investigated

Impact of different variances on decision rules:

• Biosimilar more variable left of horizontal line, reference right of line
• Monotone relationship between ratio of variances and probability of concluding

similarity, i.e. more variable reference process -> more likely to conclude similarity

• x-Sigma and TI rules often (erroneously) conclude biosimilarity even if Biosim

samples are more variable and means different. Larger variance in reference to the right

Shift in originator process

Scenario 3

Note: Illustrations use shift of +-5*SD to get a bimodal distribution

• Originator samples come from a

mixture of normal distributions with different means

• Means of originator (mixture)

distribution are a multiple of SD apart

• Positive shifts are into the direction of

the biosim process

• Negative shifts are away from the

biosim process

Impact of shift on decision rules

• Dotted line reports probability of self-similarity conclusion
• Horizontal line indicates scenario where biosimilar is equal to post-shift process
• Probability to conclude similarity is smaller compared to no shift when shift is

slightly opposite to test mean

• It increases when shift is towards test mean
• However, probability to conclude similarity for acceptance ranges based on

reference SD converges to 1 both for large shifts towards and opposite test mean

Summary & Outlook

• Dichotomy of Type I and Type II errors does not apply to an equivalence decision -

boundaries between success and Type I error are fuzzy

• Some rules (TI, X-Sigma) have undesirable properties (decreasing power with

increasing sample size, increasing error probability for shifts away from the biosim)

• We have only considered simple scenarios; have not considered:
• Alternative designs, use of historical data
• Issues with sampling (originator from the market, biosim from production process under

development)

• Multiplicity (typically there are more than one CQA)
• Sequential decision making (issues with one CQA are discredited using data from another

CQA)

BASG - Austrian Federal Office for Safety in Health Care www.basg.gv.at Traisengasse 5 1200 Vienna Florian Klinglmueller Biostatistician T + 43 (0) 50 555 36624 florian.klinglmueller@ages.at