Ma rc ia L . Zuc ke r, Ph.D. ZI VD L L C 1 I nte rpre t sta - - PowerPoint PPT Presentation

Ma rc ia L . Zuc ke r, Ph.D. ZI VD L L C

1

 I

nte rpre t sta tistic a l a na lyse s a s re po rte d b y c o mme rc ia l pro g ra ms

 I

de ntify the sta tistic a l a na lyse s re le va nt to the q ue stio n b e ing a ske d

 Critic a lly e va lua te da ta pre se nte d in

pa c ka g e inse rts fo r misuse d sta tistic s

2

 De finitio n o f Sta tistic s: T

he sc ie nc e o f pro duc ing unre lia b le fa c ts fro m re lia b le fig ure s.

 E

va n E sa r

 Be a b le to a na lyze sta tistic s, whic h c a n b e

use d to suppo rt o r unde rc ut a lmo st a ny a rg ume nt.

 Ma rilyn vo s Sa va nt  Sta tistic : a func tio n o f a se t o f o b se rva tio ns

fro m a ra ndo m va ria b le .

 CL

SI Ha rmo nize d Da ta b a se

3

 A ne w POCT

is to b e imple me nte d

› Multiple re plic a te s o f c o ntro ls run › Run side b y side pa tie nt sa mple s with c urre nt me tho d › Da ta is:

 E

nte re d into E P E va lua to r OR

 E

nte re d into Sta tisPro OR

 Se nt to ma nufa c ture r

› Re po rt re turne d with lo ts o f sta tistic s

 Re po rt ma y indic a te pa ss/ fa il to unc le a r

spe c ific a tio ns

 Ma nufa c ture r re p e xpla ins it is a ll g o o d  Ho w do I

kno w it is OK ?

4

 Adva nc e fo r Administra to rs o f the L

a b o ra to ry

 We b ina r o n sta tistic s b y Da vid Pla ut  E

xc e l T e mpla te s fo r:



L ine a rity



5 sa mple s; 2-4 re plic a te s e a c h



Re pro duc ib ility



20 va lue e va lua tio n



4 sa mple c o mpa riso n b e twe e n syste ms



Me tho d Va lida tio n



35 sa mple s



80 sa mple s  F

re e do wnlo a da b le b o o k “Unde rsta nding L a b o ra to ry Sta tistic s”



http:/ / la b o ra to ry-ma na g e r.a d va nc e we b .c o m/ We b ina r/ E d ito ria l- We b ina rs/ Ma king -Se nse -o f-L a b o ra to ry-Sta tistic s.a sp x

5

 Qua ntita tive Me tho ds

› Sta tistic s we use a ssume a norma l distribution

6

SD

 Me a sure o f the va ria b ility o f the syste m

› Ho w c lo se a re multiple re plic a te s?

 Hig he r numb e r o f re plic a te s a llo ws b e tte r

e stima te o f pre c isio n

 Outlie rs a ffe c t sma ll numb e rs muc h mo re

sig nific a ntly

 Ca lc ula tio ns a ssume a No rma l Distrib utio n

› F re q ue ntly untrue a ssumptio n, b ut use d a nywa y.

7

8

9

5 10 15 20 25 30 35 40 45 3 3.2 3.4 3.6 3.8 4 4.2 4.4 4.6 4.8 Frequency Result

N=100

1 2 3 4 5 6 3 3.25 3.5 3.75 4 4.25 Frequency Result

N=10

1 2 3 4 5 6 7 3 3.25 3.5 3.75 4 4.25 4.5 4.75 Frequency Result

N=20

 Me a n – c e ntra l te nde nc y o f the da ta

› Pe a k o f the b e ll c urve (Ave ra g e use d in pra c tic e )

 Me dia n

› Va lue whe re 50% o f sa mple s a re lo we r & 50% hig he r

 Sta nda rd de via tio n (SD) – me a sure o f va ria b ility

› Width o f the b e ll c urve › Re la te s to diffe re nc e b e twe e n individua l re sults a nd the me a n

 Sta nda rd e rro r (SE

) – me a sure o f SD o f the me a n

› Ca lc ula te d fro m va ria nc e (SD2) & N

 95% Co nfide nc e inte rva l

› E stima te o f “truth” fro m da ta c o lle c te d › 95% pro b a b ility tha t the “true ” va lue is within the inte rva l de fine d

10

11

Sta tistic

N=10 N=20 N=100

Me a n 3.90 4.17 4.22 95% CI me a n 3.65 – 4.14 4.00 – 4.35 4.14 – 4.27 SE 0.11 0.08 0.02 SD 0.34 0.38 0.24 CV = (

𝑁𝑁𝑁𝑁 𝑇𝑇 ) ∗ 100

8.7% 9.1% 5.7% Me d ia n 3.99 4.21 4.25 95% CI me d ia n 3.45 – 4.20 4.01 – 4.44 4.19 – 4.29

12

0.5 1 1.5 2 2.5 3.85 3.9 3.95 4 4.05 4.1 4.15 4.2 4.25 Frequency Result

N=8

5 10 15 20 25 3.8 3.9 4 4.1 4.2 4.3 4.4 4.5 4.6 4.7 4.8 Frequency Result

N=98

13

Statistic N=10 N=8 N=100 N=98

Me a n 3.90 4.04 4.22 4.24 95% CI me a n 3.65 – 4.14 3.92 – 4.16 4.14 – 4.27 4.20 – 4.28 SE 0.11 0.05 0.02 0.02 SD 0.34 0.14 0.24 0.20 CV = (

𝑁𝑁𝑁𝑁 𝑇𝑇 ) ∗ 100

8.7% 3.5% 5.7% 4.8% Me d ia n 3.99 4.05 4.25 4.25 95% CI me d ia n 3.45 – 4.20 3.86 – 4.23 4.19 – 4.29 4.20 – 4.30

 Sta tistic s o fte n lo o k b e tte r a t hig he r

me a n va lue s

› If me a n is 0.1 a n SD o f 0.05 is 50% CV › If me a n is 100 a n SD o f 5.0 is 5% CV

 E

va lua te va lue s re po rte d in inse rts

› Sho uld b e ne a r c linic a l de c isio n po ints › Re q uire d fo r ne we r pro duc ts › F

• r o lde r pro duc ts e xpe c t to se e mo re

va ria b ility in e nd-use r re sults

14

 Co mpa riso n to “truth”

› T ruth usua lly de fine d a s c urre nt syste m › T ruth a myth fo r ma ny a na lyte s

 No ta b ly c o a g ula tio n, tro po nin I, o the r no n-

sta nda rdize d a na lyte s

 Ho w c lo se do e s POCT

c o me to la b re sult

› Co rre la tio n using pa tie nt sa mple s

15

16

 Da ta po ints

› E a c h split sa mple g e ne ra te s o ne po int › Ho rizo nta l (X) a xis is L a b (c urre nt syste m) › Ve rtic a l (Y) a xis is po int o f c a re (ne w) de vic e

 Re g re ssio n line

› Ma the ma tic a l pre dic tio n o f re la tio nship b e twe e n two de vic e s

y = 1.03x + 3.6 R = 0.965

200 400 600 800 1000 1200

200 400 600 800 1000

ACT NE W ACT OL D

Re g re ssio n line Da ta po ints Re g re ssio n e q ua tio n

 Re g re ssio n e q ua tio n

› 3 pa rts: Y = mX + b (y = 1.03x + 3.6)

 Y = POC (ne w) re sult; X = la b (c urre nt) re sult  m = slo pe - pe rfe c t c o rre la tio n m = 1.0  b = inte rc e pt - pe rfe c t c o rre la tio n b = 0.0

› r va lue - c o rre la tio n c o e ffic ie nt

 NOT

r

2

 De sc rib e s ho w muc h o f the c ha ng e in Y va lue

is due to the c ha ng e in the X va lue

 r = 0.91 me a n 91% c o rre la tio n

 Ca nno t judg e

› All va lue s c lo se to no rma l ra ng e › No thing a b o ve 150

 E

va lua te the a xe s whe n lo o king a t c o rre la tio n g ra phs

y = 1.08x + 5.53 R = 0.906

50 60 70 80 90 100 110 120 130 140 150 50 70 90 110 130 150 POC Lab

Glucose

 Assa y ra ng e to 500, so

spre a d se e ms OK

› Iso la te d va lue drive s c o rre la tio n

 Orig ina l da ta se t sho we d

• ut o f ra ng e va lue s

› T he se must b e e xc lude d b e fo re re g re ssio n run

 Re vise d da ta ha s sa me

issue s a s prio r g luc o se re sults

21

y = 1.01x - 9.86 R = 0.980

50 100 150 200 250 300 350 400 100 200 300 400

y = 0.94x - 1.90 R = 0.937

20 40 60 80 100 120 140 160 180 50 100 150 200

 Da ta ne e d to spa n the c linic a lly

impo rta nt ra ng e

› Sing le e xtre me va lue s sho uld b e o mitte d › Out o f ra ng e va lue s must b e o mitte d

22

 Diffe re nc e plo t

› Bla nd Altma n a na lysis › Plo t e ithe r re fe re nc e re sult o r a ve ra g e o f two me tho ds a s X

 Re fe re nc e re sult use d whe n c o nside re d “truth”

 e .g ., POC e le c tro lyte s ve rsus la b

 Ave ra g e use d whe n “truth” is unc e rta in

 e .g ., ACT

c o mpa riso ns

› Plo t diffe re nc e b e twe e n two re sults a s Y

23

 L

• o k fo r b ia s

› Co nsta nt o r va ria b le ? › Clinic a lly sig nific a nt?

• 6.0
• 5.0
• 4.0
• 3.0
• 2.0
• 1.0

0.0 1.0 2.0 3.0 4.0 5.0 6.0 0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 10.0 Current INR - New INR Mean INR

• 6.0
• 5.0
• 4.0
• 3.0
• 2.0
• 1.0

0.0 1.0 2.0 3.0 4.0 5.0 6.0 0.0 2.0 4.0 6.0 8.0 10.0 Current INR - New INR Mean INRs

 Cha ng e o f c linic a l de c isio n limit c a n

ma inta in c urre nt pra c tic e sta nda rds

25

y = 1.09x - 7.53 R = 0.915

100 200 300 400 500 600 700 800 900 200 400 600 800 1000

Ne w ACT Cur r e nt ACT

T a rg e t T ime c ha ng e fro m 480 to 520 se c o nds

LAB POC A >0.1 <0.1 >0.1 28 1 PPV 97% <0.1 2 9 NPV 82% Sensitivity Specificity Concordance 93% 90% 93% LAB POC B >0.1 <0.1 >0.1 18 PPV 100% <0.1 12 10 NPV 45% Sensitivity Specificity Concordance 60% 100% 70%

26

 Se nsitivity › a b ility o f a n a ssa y to ide ntify pa tie nts with a spe c ific c o nditio n (true po sitive s)  Spe c ific ity › a b ility o f a n a ssa y to ide ntify pa tie nts witho ut a spe c ific c o nditio n (true ne g ative s)  Po sitive pre dic tive va lue › like liho o d tha t a pa tie nt with a po sitive re sult (o r a b o ve the c ut-o ff) truly ha s the c o nditio n  Ne g a tive pre dic tive va lue › like liho o d tha t a pa tie nt with a ne g a tive re sult (o r b e lo w the c ut-o ff) is truly no rma l

27

“T rue ” Re sult Positive Ne gative Ne w Syste m Re sult Positive

T rue po sitive (T P) F a lse po sitive (F P) Po sitive pre d ic tive va lue (PPV)

Ne gative F

a lse ne g a tive (F N) T rue ne g a tive (T N) Ne g a tive pre d ic tive va lue (NPV) Se nsitivity Spe c ific ity Co nc o rd a nc e

𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇 = 𝑈𝑈 𝑈𝑈 + 𝐺𝐺 𝑈𝑈𝑄 = 𝑈𝑈 𝑈𝑈 + 𝐺𝑈

28

𝑇𝑞𝑇𝑞𝑇𝑞𝑇𝑞𝑇𝑇𝑇 = 𝑈𝐺 𝑈𝐺 + 𝐺𝑈 𝐺𝑈𝑄 = 𝑈𝐺 𝑈𝐺 + 𝐺𝐺 𝐷𝐷𝑇𝑞𝐷𝐷𝐷𝐷𝑇𝑞𝑇 = 𝑈𝑈 + 𝑈𝐺 𝑈𝐷𝑇𝐷𝑈 𝑇𝐷𝑇𝑞𝑈𝑇 𝐺𝑂𝑇𝑂𝑇𝐷

 Qua lita tive te sts a lwa ys inc lude se nsitivity

a nd spe c ific ity c la ims

› Olde r pro duc ts ha ve limite d c linic a l da ta

 Only spike d sa mple s e va lua te d  Only fro ze n c linic a l sa mple s e va lua te d  T

• o fe w sa mple s e va lua te d

› Ne we r pro duc ts will inc lude c o nfide nc e inte rva ls

 Do no t wa nt te st whe re CI

spa ns 50% (c o in to ss)

29

 Pa ire d t-te st › Co mpa re the diffe re nc e b e twe e n pa ire d sa mple s › Null hypo the sis is te ste d  me a n diffe re nc e is ze ro › Me a ns o f po pula tio ns c o mpa re d › Assume no rma l distrib utio n; e q ua l va ria nc e  ANOVA (Ana lysis o f Va ria nc e ) › Co mpa re me a ns o f g ro ups o f me a sure me nt › Null hypo the sis is te ste d  me a ns o f the me a sure d va ria b le s a re the sa me › Va ria nc e s o f po pula tio ns c o mpa re d › Assume no rma l distrib utio n; e q ua l va ria nc e

30

 Sta tistic a l sig nific a nc e c a n b e de fine d a t

multiple le ve ls

 F

• r dia g no stic s, g e ne ra lly de fine d a s

p < 0.05

› 95% c o nfide nc e › ~ + 2 SD fro m me a n

31

 If vie wing re sults o f a na lysis:

› p < 0.05 two sa mple s a re diffe re nt › 0.05 < p < 0.1 ? tre nd to wa rds diffe re nc e › p > 0.1 two sa mple s a re the sa me

32

 T

he re a re a s ma ny wa ys to c runc h da ta a s the re a re pe o ple to do it.

 K

e e p in mind wha t yo u a re lo o king fo r

› Clinic a l utility

 sta tistic a l diffe re nc e ma y no t ma tte r

 Unde rsta nd wha t yo u wa nt BE

F ORE yo u c o lle c t

the da ta

› De fine studie s b y the info rma tio n yo u wa nt

33

 T

he re a re thre e kinds o f lie s: lie s, da mne d lie s a nd sta tistic s.

 Be nja min Disra e li  T

• rture numb e rs, a nd the y’ ll c o nfe ss to

a nything .

 Gre g g E

a ste rb ro o k

34

Ma rc ia L . Zuc ke r, Ph.D. ZI VD, L L C mlzuc ke r@ ve rizo n.ne t

35