E xpe rime nts De sig n a nd Ana lysis
F
- tis E
. Pso mo po ulo s
CODAT A-RDA Advanc e d Bio info rmatic s Wo rksho p, 20-24 Aug ust 2018, T rie ste , I taly
E xpe rime nts De sig n a nd Ana lysis F o tis E . Pso mo po ulo - - PowerPoint PPT Presentation
E xpe rime nts De sig n a nd Ana lysis F o tis E . Pso mo po ulo s CODAT A-RDA Advanc e d Bio info rmatic s Wo rksho p, 20-24 Aug ust 2018, T rie ste , I taly A sho rt intro to me 2 Bio info rma tic s a nd Da ta Mining
F
. Pso mo po ulo s
CODAT A-RDA Advanc e d Bio info rmatic s Wo rksho p, 20-24 Aug ust 2018, T rie ste , I taly
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
2 Bio info rma tic s Clo ud Co mputing Da ta Mining
Bio info rma tic s a nd Da ta Mining
to o ls a nd pipe line s to a ddre ss
do ma in-spe c ific q ue stio ns
Bio info rma tic s a nd Clo ud
Co mputing
wo rkflo ws a nd pipe line s o n
c lo ud infra struc ture s
sa yo po ulo u, PhD Stud e nt
intsa kis, PhD Stud e nt
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
3
CE RT H Main Building
NGS Da ta Ana lysis using Clo ud Co mputing (Oc t 2015) 1st So ftwa re Ca rpe ntry Wo rksho p (Oc t 2016)
NGS Wo rkflo ws Omic s Da ta I
nte g ra tio n
Da ta Mining
T ra ining Re se a rc h Pe o ple
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
4
Go to www.me nti.c om and use the c ode 45 89 48
xpe rime nts a llo w us to se t up a dire c t c o mpa riso n b e twe e n the tre a tme nts
a llo ws us to ma ke stro ng e r infe re nc e s a b o ut the na ture o f diffe re nc e s tha t we se e in the e xpe rime nt. Spe c ific a lly, we ma y ma ke infe re nc e s a b o ut c a usa tio n.
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
5 An e xpe rime nt is c ha ra c te rize d b y the tr
e atme nts a nd e xpe r ime ntal units to b e use d, the wa y tre a tme nts a re assigne d to units, a nd the r e sponse s tha t a re me a sure d.
An a lte rna tive de finitio n is:
“tre a tme nt de sig n” is the se le c tio n o f tre a tme nts to b e use d “e xpe rime nt de sig n” is the se le c tio n o f units a nd a ssig nme nt o f tre a tme nts
No te tha t the re is no me ntio n o f a me tho d fo r a na lyzing the re sults.
a na lysis is not pa rt o f the de sig n Ho we ve r: it is o fte n use ful to c o nside r the a na lysis whe n pla nning a n e xpe rime nt.
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
6 T re a tme nts, units, a nd a ssig nme nt me tho d spe c ify the e xpe rime nta l de sig n
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
7
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
8
http://ne ilfws.g ithub .io /Pub Me d/pmre trac t/pmre trac t.html
Cost o f e xpe rime nta tio n.
We ha ve a re spo nsib ility to do no rs!
L
imite d & Pr e c ious ma te ria l
e sp. c linic a l sa mple s.
Immor
talization o f da ta se ts in pub lic da ta b a se s a nd me tho ds in the
lite ra ture . Our b a d sc ie nc e b e g e ts mo re b a d sc ie nc e .
E
thic al c onc e r ns o f e xpe rime nta tio n: a nima ls a nd c linic a l sa mple s.
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
9
Slide s adapte d fro m “De sig ning F unc tio nal Ge no mic s E xpe rime nts fo r Suc c e ssful Analysis”, b y Ro ry Stark, 18/09/2017, CRUK-CI
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
10
Go to www.me nti.c om and use the c ode 45 89 48
No t a ll e xpe rime nta l de sig ns a re c re a te d e q ua l! A g o o d e xpe rime nta l de sig n must
1. Avo id syste ma tic e rro r 2. Be pre c ise 3. Allo w e stima tio n o f e rro r 4. Ha ve b ro a d va lidity
L
e t’ s se e the se a spe c ts o ne a t a time !
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
11
Slide s adapte d fro m Gary W. Oe hle rt, “A F irst Co urse in De sig n and Analysis o f E xpe rime nts”, 2010 - I SBN 0-7167-3510-5
Co mpa ra tive e xpe rime nts e stima te diffe re nc e s in re spo nse b e twe e n
tre a tme nts.
I
f a n e xpe rime nt ha s syste ma tic e rro r, the n the c o mpa riso ns will b e b ia se d, no ma tte r ho w pre c ise o ur me a sure me nts a re o r ho w ma ny e xpe rime nta l units we use .
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
12
I f re spo nse s fo r units re c e iving tr
e atme nt one a re me a sure d with instr ume nt A
a nd re spo nse s fo r tr
e atme nt two a re me a sure d with instr ume nt B,
the n we do n’ t kno w if a ny o b se rve d diffe re nc e s a re due to tre a tme nt e ffe c ts o r instrume nt misc a lib ra tio ns.
E
ve n witho ut syste ma tic e rro r, the re will b e ra ndo m e rro r in the re spo nse s, a nd this will le a d to ra ndo m e rro r in the tre a tme nt c o mpa riso ns.
E
xpe rime nts a re pre c ise whe n this ra ndo m e rro r in tre a tme nt c o mpa riso ns is sma ll.
Pre c isio n de pe nds o n the size o f the ra ndo m e rro rs in the re spo nse s, the
numb e r o f units use d, a nd the e xpe rime nta l de sig n use d.
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
13
E
xpe rime nts must b e de sig ne d so tha t we ha ve a n e stima te o f the size o f ra ndo m e rro r.
T
his pe rmits sta tistic a l infe re nc e :
fo r e xa mple , c o nfide nc e inte rva ls o r te sts o f sig nific a nc e .
We c annot do infe r
e nc e without an e stimate of e r r
! Sa dly, e xpe rime nts
tha t c a nno t e stima te e rro r c o ntinue to b e run.
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
14
We will se e those in pra c tic e la te r.
T
he c o nc lusio ns we dra w fro m a n e xpe rime nt a re a pplic a b le to the e xpe rime nta l units we use d in the e xpe rime nt.
I
f the units a re a c tua lly a sta tistic a l sa mple fro m so me po pula tio n o f units, the n the c o nc lusio ns a re a lso va lid fo r the po pula tio n.
Be yo nd this, we a re e xtra po la ting , a nd the e xtra po la tio n mig ht o r mig ht no t
b e suc c e ssful.
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
15
We c o mpa re two diffe re nt drug s fo r tre a ting a tte ntio n de fic it diso rde r a nd o ur sub je c ts a re pr
e - adole sc e nt boys fro m our c linic .
b ut e ve n tha t mig ht no t b e true if o ur c linic ’ s po pula tio n o f sub je c ts is unusua l in so me wa y.
he re sults a re e ve n le ss c o mpe lling fo r o lde r b o ys o r fo r g irls.
re a tme nts
xpe rime nta l units
a c to rs
xpe rime nta l E rro r 10.Blinding
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
16
1. T r e atme nts a re the diffe re nt pro c e dure s we wa nt to c o mpa re . diffe re nt kinds o r a mo unts o f fe rtilize r in a g ro no my diffe re nt lo ng dista nc e ra te struc ture s in ma rke ting diffe re nt te mpe ra ture s in a re a c to r ve sse l in c he mic a l e ng ine e ring 2. E xpe r ime ntal units a re the thing s to whic h we a pply the tre a tme nts. plo ts o f la nd re c e iving fe rtilize r g ro ups o f c usto me rs re c e iving diffe re nt ra te struc ture s b a tc he s o f fe e dsto c k pro c e ssing a t diffe re nt te mpe ra ture s
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
17
3. Re sponse s a re o utc o me s tha t we o b se rve a fte r a pplying a tre a tme nt to a n
e xpe rime nta l unit (a me a sure o f wha t ha ppe ne d in the e xpe rime nt; we o fte n ha ve mo re tha n o ne re spo nse )
nitro g e n c o nte nt o r b io ma ss o f c o rn pla nts pro fit b y c usto me r g ro up yie ld a nd q ua lity o f the pro d uc t pe r to n o f ra w ma te ria l 4. Me a sur e me nt units (o r re spo nse units) a re the a c tua l o b je c ts o n whic h the
re spo nse is me a sure d. T he se ma y diffe r fro m the e xpe rime nta l units.
(e .g . in d iffe re nt fe rtilize rs o n the nitro g e n c o nte nt o f c o rn pla nts) Diffe re nt fie ld plo ts
a re the e xpe rime nta l units, b ut the me a sure me nt units mig ht b e a sub se t o f the c o rn pla nts o n the fie ld plo t, o r a sa mple o f le a ve s, sta lks, a nd ro o ts fro m the fie ld plo t.
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
18
5. Ra ndomiza tion is the use o f a kno wn, unde rsto o d pro b a b ilistic me c ha nism fo r
the a ssig nme nt o f tre a tme nts to units.
Othe r a spe c ts o f a n e xpe rime nt c a n a lso b e ra nd o mize d : fo r e xa mple , the o rd e r in
whic h units a re e va lua te d fo r the ir re spo nse s.
6. Contr
An e xpe rime nt is c o ntro lle d b e c a use we a s e xpe rime nte rs a ssig n tre a tme nts to
e xpe rime nta l units. Othe rwise , we wo uld ha ve a n o b se rva tio na l stud y.
A c o ntro l tre a tme nt is a “sta nd a rd ” tre a tme nt tha t is use d a s a b a se line o r b a sis o f
c o mpa riso n fo r the o the r tre a tme nts.
T
his c o ntro l tre a tme nt mig ht b e the tre a tme nt in c o mmo n use , o r it mig ht b e a null tre atme nt (no tre a tme nt a t a ll).
e .g . a study o n the e ffic a c y o f fe rtilize r c o uld g ive so me fie lds no fe rtilize r a t a ll.
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
19
7. F ac tor s c o mb ine to fo rm tre a tme nts. the b a king tre a tme nt fo r a c a ke invo lve s a g ive n time a t a g ive n te mpe ra ture . T he tre a tme nt is the c o mb ina tio n o f time a nd te mpe ra ture , b ut we c a n va ry the time a nd te mpe ra ture se pa ra te ly. T hus we spe a k o f a time fa c to r a nd a te mpe ra ture fa c to r. I ndividua l se tting s fo r e a c h fa c to r a re c a lle d le ve ls o f the fa c to r. 8. Confounding o c c urs whe n the e ffe c t o f o ne fa c to r o r tre a tme nt c a nno t b e
disting uishe d fro m tha t o f a no the r fa c to r o r tre a tme nt.
E
xc e pt in ve ry spe c ia l c irc umsta nc e s, c o nfo unding sho uld b e a vo ide d.
e .g . pla nting c o rn va rie ty A in Minne so ta a nd c o rn va rie ty B in I
n this e xpe rime nt, we c a nno t disting uish lo c a tio n e ffe c ts fro m va rie ty e ffe c ts—the va rie ty fa c to r a nd the lo c a tio n fa c to r a re c o nfo unde d.
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
20
9. E xpe r ime nta l E r r
is the ra ndo m va ria tio n pre se nt in a ll e xpe rime nta l re sults. Diffe re nt e xpe rime nta l units will g ive d iffe re nt re spo nse s to the sa me tre a tme nt, a nd it
is o fte n true tha t a pplying the sa me tre a tme nt o ve r a nd o ve r a g a in to the sa me unit will re sult in d iffe re nt re spo nse s in d iffe re nt tria ls.
E
xpe rime nta l e rro r d o e s no t re fe r to c o nd uc ting the wro ng e xpe rime nt o r d ro pping te st tub e s.
tre a tme nt wa s g ive n to whic h unit.
he lps pre ve nt b ia s in the e va lua tio n, e ve n unc o nsc io us b ia s fro m we ll-inte ntio ne d
e va lua to rs.
Do ub le b lind ing o c c urs whe n b o th the e va lua to rs o f the re spo nse a nd the (huma n
sub je c t) e xpe rime nta l units d o no t kno w the a ssig nme nt o f tre a tme nts to units.
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
21
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
22
Sho uld ha ve :
1. Cle a r Ob je c tive s 2. F
3. Suffic ie nt Po we r 4. Ra nd o mize d Co mpa riso ns
And b e :
1. Pre c ise 2. Unb ia se d 3. Ame na b le to sta tistic a l a na lysis 4. Re pro d uc ib le
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
23
So und re aso nab le … Ho w are the se applie d in prac tic e tho ug h?
E
xpe rime nta l F a c to rs
Va ria b ility
1. So urc e s o f Va ria nc e 2. Re plic a te s
Bia s
1. Co nfo unding fa c to rs 2. Ra ndo miza tio n whe re ve r a de c isio n is to b e ma de
3. Co ntro ls
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
24
F a c to rs: Aspe c ts o f E xpe rime nt tha t c ha ng e a nd influe nc e the o utc o me o f the e xpe rime nt
e .g . time , we ig ht, d rug , g e nd e r, e thnic ity, c o untry, pla te , c a g e e tc .
Va ria b le type de pe nds o n type o f me a sure me nt
Ca te g o ric a l (nominal) , e .g . g e nd e r
Ca te g o ric a l with o rd e ring (or
dinal), e .g . tumo r g ra d e
Disc r e te, e .g . sho e size , numb e r o f c e lls
Continuous, e .g . b o d y we ig ht in kg , he ig ht in c m
I nde pe nde nt o r De pe nde nt Va ria b le s
Ind e pe nd e nt va ria b le (IV): wha t you c ha ng e
De pe nd e nt va ria b le (DV): wha t c ha ng e s due to IV
“I f (inde pe nde nt variab le ), the n (de pe nde nt variab le )”
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
25
Bio lo g ic a l “No ise ”
Bio lo g ic a l pro c e sse s a re inhe re ntly sto c ha stic Sing le c e lls, c e ll po pula tio ns, individua ls, o rg a ns, spe c ie s…. T
ime po ints, c e ll c yc le , sync hro nize d vs. unsync hro nize d
T
e c hnic a l No ise
Re a g e nts, a ntib o die s, te mpe ra ture s, po llutio n Pla tfo rms, runs, o pe ra to rs
Co nside r in a dva nc e a nd c o ntro l re plic a tio n re q uire d to c a pture va ria nc e
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
26
Bio lo g ic a l Re plic a tio n
In vivo
Pa tie nts Mic e
In vitro
Diffe re nt c e ll line s Re -g ro wing c e lls (pa ssa g e s)
T
e c hnic a l Re plic a tio n
E
xpe rime nta l pro to c o l
Me a sure me nt pla tfo rm (i.e . se q ue nc e r)
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
27
Ca lc ula ting a ppro pria te sa mple size s Po we r c a lc ula tio ns Pla nning fo r pre c isio n Re so urc e e q ua tio n
Po we r: the pr
I
de ntify a nd c o ntro l the so urc e s o f va ria b ility
Bio lo g ic a l va ria b ility T
e c hnic a l va ria b ility
Using a ppro pria te numb e rs o f sa mple s (sa mple size / re plic a te s) Po we r c a lc ula tio ns e stima te sa mple size re q uire d to de te c t a n e ffe c t if de g re e o f variab ility
is kno wn
Depends on δ, n, sd , a , H
A
I
f a dding sa mple s inc re a se s va ria b ility, tha t a lo ne wo n’ t a dd po we r!
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
28
Aka E xtra ne o us, hidde n, lurking o r ma sking fa c to rs, o r the third va ria b le o r me dia to r va ria b le .
Ma y ma sk a n a c tua l a sso c ia tio n o r fa lse ly de mo nstra te a n a ppa re nt a sso c ia tio n b e twe e n the inde pe nde nt a nd de pe nde nt va ria b le s.
Hypo the tic a l e xa mple wo uld b e a study o f c o ffe e drinking a nd lung c a nc e r.
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
29
Othe r fa c tor
smo king (c o nfo unding variab le )
E ffe c t/ Outc ome
L ung c a nc e r (de pe nde nt variab le )
F AL SE ASSOCIAT ION
Ca use
Drinking Co ffe e (inde pe nde nt variab le )
I
na de q ua te ma na g e me nt a nd mo nito ring o f c o nfo unding fa c to rs
o ne o f the mo st c o mmo n c a use s o f re se a rc he rs wro ng ly a ssuming tha t a
c o rre la tio n le a ds to a c a usa lity.
I
f a study do e s no t c o nside r c o nfo unding fa c to rs, do n’ t b e lie ve it!
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
30
Ra ndo miza tio n
Sta tistic a l a na lysis a ssume ra ndo mize d c o mpa riso ns Ma y no t se e issue d c a use d b y no n-ra ndo mize d c o mpa riso ns Ma ke e ve ry de c isio n ra ndo m no t a rb itra ry
Blinding
E
spe c ia lly impo rta nt whe re sub je c tive me a sure me nts a re ta ke n
E
ve ry e xpe rime nt sho uld re a c h its po te ntia l de g re e o f b linding
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
31
RNA E
xtra c tio n
T
he diffe re nc e b e twe e n Co ntro l, T re a tme nt 1 a nd T re a tme nt 2 is c o nfo unde d b y da y a nd pla te .
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
32
Contr
T r e atme nt 1 T r e atme nt 2 Day 1, Plate 1 Day 2, Plate 2 Day 3, Plate 3
Blo c king is the a rra ng ing o f e xpe rime nta l units in g ro ups (b lo c ks) tha t a re simila r to o ne a no the r
RBD a c ro ss pla te s so tha t e a c h pla te c o nta ins spa tia lly ra nd o mize d e q ua l pro po rtio ns o f:
1. Co ntro l 2. T re a tme nt 1 3. T re a tme nt 2
c o ntro lling pla te e ffe c ts.
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
33
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
34
te c hnic a l? Ho w ma ny sa mple s/ re plic a te s sho uld b e c o lle c te d?
Whic h g e ne se t(s) will yo u use fo r pa thwa y a na lysis?
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
35
10.Wha t info rma tio n a b o ut yo ur e xpe rime nt sho uld b e re c o rde d to he lp ide ntify a ny pro b le ms sho uld the re b e a ny? 11.Will yo u b e multiple xing sa mple s? Ho w will yo u a ssig n b a rc o de s? Will yo u use po o le d lib ra rie s? Ho w ma ny po o ls? Ho w will sa mple s b e a ssig ne d to po o ls? 12.Wha t a re the se q ue nc ing pa ra me te rs yo u ne e d to b e a wa re o f (e .g . se q ue nc ing type a nd de pth)? 13.Wha t o the r type s o f da ta mig ht b e use ful to a ssa y, a nd ho w mig ht the se q ue nc ing pa ra me te rs ne e d to c ha ng e to a c c o mmo da te this? 14.Ca n yo u think o f a ny o the r de sig n re la te d issue s tha t c o uld/ sho uld b e a ddre sse d?
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
36
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
37
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
38
150 individua ls 50 o f e a c h tre a tme nt T
re a tme nt la sts 1 we e k
We ha ve 3 inc ub a to rs/ g re e nho use s/ ta nks/ c a g e s whic h e a c h ho ld 50
individua ls
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
39
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
40
We e k 1 We e k 2 We e k 3
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
41
We e k 1 We e k 2 We e k 3
L
e t’ s do the b lue tre a tme nt in we e k 1, g re e n tre a tme nt in we e k 2 a nd re d tre a tme nt in we e k 3
b e c a use … re a so ns!
Yo u ha ve 3 unde rg ra ds. Ho w sho uld the y split the da ta c o lle c tio n wo rk? T
he y a re a lso a va ila b le fo r just two da ys to do the lib ra ry pre p.
And! Yo u just ha ve 2 la ne s pe r Se q ue nc e r a va ila b le
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
42
Disc uss in gr
Mo nda y, Aug ust 20th 2018 E xpe rime nt De sig n a nd Ana lysis
43