. logit - - PDF document

• IndianaUniversity
• PrinciplesofWorkflowin

DataAnalysis

ScottLong

• November2010
• 1.Acoordinatedframeworkforconductingdataanalysis

2.WFinvolvescoordinatedproceduresfor:

• Planning,organizinganddocumentingresearch
• Cleaningdata
• Analyzingdata
• Presentingresults
• Backingupandarchivingmaterials
• 1.YourWFmightbe:

A.Plannedandcarefullyorchestrated. B.Adhoc,piecemeal,developedinreactiontomistakes. 2.YoucanimproveyourWFwithamodestinvestmentoftime. A.Thelessexperienceyouhave,theeasieritis. B.Itwillsaveyoutimeandmakeyouabetterdataanalyst.

• 1.Replication
• Replicationisessentialforgoodscience.
• Aneffectiveworkflowisessentialforreplication.

2.Gettingtherightanswers

• Retractionsareembarrassingandcanendcareers.

3.Time

• “Scienceisavoraciousinstitution.”
• Aneffectiveworkflowmakesyoumoreefficient.

4.Errorsareinevitable;aneffectiveworkflowhelpsyoufindandfixthem.

• 5.GainingtheIUadvantage
• “Thepublicationof[TheWorkflowofDataAnalysis

UsingStata]mayevenreduceIndiana’scomparative advantageofproducinghotshotquantPhDsnowthat gradstudentselsewherecanvicariouslybenefitfrom thisimportantaspectofthetrainingthere.”Gabriel Rossmanonhisblog

• 1.Easythings:consultingoneasythings,insteadofhardthings.

2.Incorrectresultswithclever“explanations”. 3.Adissertationdelayed18monthstodeterminewhyresultschanged. 4.Irreproducibleresultsfromasingle,743linedofile. 5.Analyzingthewrongdataset:“Thedatasetsareexactlythesameexcept thatIchangedthemarriedvariable.” 6.AnalyzingthewrongvariablewhilewritinganNASreport. 7.Miscodedgenesthatdelayedprogressinastudyofalcholism. 8.Collaborationsthatmultiplythewaysthingscangowrong. 9.Misleadingorambiguousoutputsuchas...

• Example 1: definitely a problem in a $3M study

. tabulate female sdchild_v1 R is | Q15 Would let X care for children female? | Defintel Probably Probably Definitel | Total

• ---------+---------------------------------------------+----------

0Male | 41 99 155 197 | 492 1Female | 73 98 156 215 | 542

• ---------+---------------------------------------------+----------

Total | 114 197 311 412 | 1,034

• Example 2: which number is which?

. tab occ ed, row | Years of education Occupation | 3 6 7 8 9 10 11 12 13 | Total

• ----------+-------------------------------------------------------------------------------
• -------------------+----------

Menial | 0 2 0 0 3 1 3 12 2 | 31 | 0.00 6.45 0.00 0.00 9.68 3.23 9.68 38.71 6.45 | 100.00

• ----------+-------------------------------------------------------------------------------
• -------------------+----------

BlueCol | 1 3 1 7 4 6 5 26 7 | 69 | 1.45 4.35 1.45 10.14 5.80 8.70 7.25 37.68 10.14 | 100.00

• ----------+-------------------------------------------------------------------------------
• -------------------+----------

Craft | 0 3 2 3 2 2 7 39 7 | 84 | 0.00 3.57 2.38 3.57 2.38 2.38 8.33 46.43 8.33 | 100.00

• ----------+-------------------------------------------------------------------------------
• -------------------+----------

WhiteCol | 0 0 0 1 0 1 2 19 4 | 41 | 0.00 0.00 0.00 2.44 0.00 2.44 4.88 46.34 9.76 | 100.00

• ----------+-------------------------------------------------------------------------------
• -------------------+----------
• Example 3: good software doing things badly

. logit tenure i.female i.female#c.articles i.male i.male#c.articles, nocons note: 0.male#c.articles omitted because of collinearity note: 1.male#c.articles omitted because of collinearity

• tenure | Coef. Std. Err. z P>|z| [95% Conf. Interval]
• ------------+----------------------------------------------------------------

1.female | -2.473265 .1351561 -18.30 0.000 -2.738166 -2.208364 | female#| c.articles | 0 | .0980976 .0098808 9.93 0.000 .0787316 .1174636 1 | .0421485 .0098962 4.26 0.000 .0227524 .0615447 | 1.male | -2.693147 .1170916 -23.00 0.000 -2.922642 -2.463651 | male#| c.articles | 0 | (omitted) 1 | (omitted)

• DidStataCorpreadtheWFbook?
• 1.Tacitknowledge

2.Heavylifting 3.Timetopractice

• 1.Explicitknowledgeisthestuffoftextbooksandarticles.
• 2.Tacitknowledgeisimplicitandundocumented(MichaelPolanyi).

A. Peopleareunawareoftheiressentialtacitknowledge.

• HenryBessemer’spatentformakingsteeldidn’twork(1855)

B.Tacitknowledgeistransferred“atthebench”.

• Personalcomputersimpedethetransferoftacitknowledge.
• Data analysis includes a lot of heavy lifting

“Thereality,ofcourse,todayisthatifyoucomeupwithagreatideayou don'tgettogoquicklytoasuccessfulproduct.There'salotof undifferentiatedheavyliftingthatstandsbetweenyourideaandthat success.”JeffBezos,amazon.com

• TheWorkflowofDataAnalysisUsingStata
• 1.MakestacitknowledgeaboutWFexplicit.
• 2.Itdealswithalotofundifferentiatedheavylifting.
• 3.Itcontainsspecificsonthegeneralissuesdiscussedtoday.
• 4.ThebookfocusesontoolsinStata,buttheprinciplesapplybroadly.
• ironicaloptimism

Theuniversalaptitudeforineptitudemakesanyhumanaccomplishment anincrediblemiracle.Dr.JohnPaulStapp

• replication

1.AneffectiveWFfacilitatesreplication. 2.Youmustplanforreplicationatthestartofaproject. 3.Disciplinesareincreasinglyconcernedwithreplicability.

• ArticlesinPoliticalScience,Economics,Sociologyandotherfields.

4.Askyourself:

• Areyourdofilesandlogfilesreadyforpublicdisplay?
• Willtheyproduceexactlythesameresultsasyouhavepublished?
• 1.Thecurseofdimensionality:10minordecisions,leadsto1,024reasonable

waystocreateyourdata.

• Wheretotruncateavariable.
• TheseedfortheRNgenerator.
• Creatingascalewithpartialmissingdata.
• Whichcasestokeepforanalysis.
• Howtocodeeducation?
• Whatvaluestoassignincomegreaterthan$200,000?
• Andsoon...
• Decisions in the path to analysis: the choices that could be made
• Decisions in the path to analysis: the choices made

• 2.Documentation:Replicationshouldinvolveretrievingdocumentation,not

tryingtorememberwhatyoudid. 3.Changingsoftware:2weeksofsleeplessnightsduetoversionvariation.This isparticularlydifficultwhenthereisanactiveusercommunity. 4.Lostfiles:corrupted,lost,unreadable,obsolete,orambiguousfiles.

• Criteria
• Ifyourprogramisnotcorrect,thennothingelsematters.
• OliveiraandStewart
• Completingworkquicklygivenaccuracyandreplicability.
• Tensionbetweenworkingquicklyandworkingcarefully.
• Don'trepeatedlyandinconsistentlydecidehowtodothings.
• Standardizationmakesiteasiertofindmistakes.
• Automatedprocedurespreventmistakesandarefaster.
• Drukker'sDictim:Nevertypeanythingthatyoucanobtainfromasaved

result.(Didtheauthorsofmarginsthinkaboutthis?)

• Themorecomplicatedyourproceduresthemorelikelyyouwillmake

mistakesorabandonyourplan.

• Yourworkflowshouldreflectthewayyou liketowork.
• Ifyouignoreyourprocedures,itisnotagoodWF.
• Differentprojectsrequiredifferentworkflows.
• Collaborationmakesitmoredifficulttohaveaneffective,efficientand

replicableworkflow.

• Why?And,whycan’ttheydoitjustlikeme?
• Everyproblemyoucanhaveworkingbyyourselfismultipled.

• 1.Agreeduponstandards

2.Explicitcoordination 3.Enforcementofstandards 4.Asenseofhumor

• Steps
• Datamustbeaccurate.
• Variablesmustbecarefullynamedandlabeled.
• Thistakes90%ofthetime,unlessyouhurry.
• Estimatemodelsandcreategraphs.
• Oftenthesimplestpartoftheworkflow.
• Incorporateoutputintoyourpresentation.
• Maintaintheprovenanceofresults.
• Makeeffectivepresentations.
• Backingupandarchiving:preservingthebitsandthecontent.

$2,000toget1variablefroman“archived”file.

• Replicationisimpossiblewithoutyourdataanddofiles.
• "Today'snoiseistomorrow'sknowledge."DavidClemmer

• Tasks
• The ideal

BlauandDuncan(1967)TheAmericanOccupationalStructure

• Allanalyseswerespecified9monthsbeforeoutputwasreceived.
• Thebookwaswrittenbasedentirelyonthoseanalyses.
• Noneofthelaterbookswrittenwithfullaccesstothedatawereasgood.
• Issues in planning

1.Aplanisaremindertostayontrack,finishtheproject,andpublishresults. Work.Finish.Publish.MichaelFaraday’ssigninhislab 2.Alittleplanninggoesalongwayandalmostalwayssavestime. 3.Planningincludes:

• Generalgoals,publishingplans,andfirmdeadlines.
• Divisionoflaborandaccountability.
• Proposalfordataconstruction:names,labels,formats.
• Proceduresforhandlingmissingdata.
• Anticipatedanalyses.
• Guidelinesandresponsibilityfordocumentation.
• Proceduresandscheduleforbackingupandarchivingmaterials.
• 1.Organizationismovtivatedbytheneedto:
• Findthings
• Avoidduplication

2.Itrequiresexplicit,consistentdecisionsaboutnamingandstoring things. 3.Organization:

• Helpsyouworkfaster
• Rewardsconsistencyanduniformity
• Organizationiscontagious
• 1.Youcan'tfindafileandthinkyoudeletedit.

2.Youfindmultipleversionsofafileanddon'tknowwhichiswhich. 3.Youandacolleagueareworkingondifferentversionsofthesamepaper. Youchangedwhatshechangedandnowyouhavethreeversionsofthe paper. 4.Youneedthefinalversionofthepaperthewassubmittedforreview,but youhavetwo(or16)fileswith"final"inthename.

• final_report_v16.docx
• NSF_science_report20101021.docx
• 1.Itiseasiertocreateafilethantofindafile.

2.Itiseasiertofindafilethantoknowwhatisinthefile. 3.Withdiskspacesocheap,itistemptingtocreatealotoffiles.

• Organizing: a standard directory structure for all projects

\WF project \- History \2009-03-06 project directory created \- Hold then delete \- Pre posted \- To clean \Documentation \Posted \Resources \Text \- Versions \Work \- To do

• Forexample,abatchfilemakescreatinguniformdirectorieseasy.
• Organizing: wfsetupsingle.bat makes it easy

REM workflow talk 2 \ wfsetupsingle.bat jsl 2009-07-12 REM directory structure for single person. FOR /F "tokens=2,3,4 delims=/- " %%a in ("%DATE%") do set CDATE=%%c-%%a-%%b md "- History\%cdate% project directory created" md "- Hold then delete " md "- Pre posted " md "- To clean" md "Documentation" md "Posted" md "Resources" md "Text\- Versions\" md "Work\- To do"

• capture log close

log using wftalk-example, replace text // program: wftalk-example.do // task: // project: // author: jsl \ 2010-07-27 version 11 clear all set linesize 80 local tag "wftalk-example.do jsl 2010-07-27" // #1 // Description of task 1 // #2 // Description of task 2

log close exit

• Templatesmakethisstructureeasytouse.
• Anycoloryouwantaslongasitisblack….
• 1.Long'sLaw:Itisalwaysfastertodocumentittodaythantomorrow.

Corollary1:Nobodylikestowritedocumentation. Corollary2:Nobodyregretshavingwrittendocumentation. Haveyoueversaid:"Drat,thisprogramhastoomanycomments." 2.Documentationoccursonmanylevels:logs,metadata,comments,names. 3.Withoutdocumentation,replicationisvirtuallyimpossible,mistakesare morelikely,andworktakeslonger. 4.Themorecodifiedthefieldthegreatertheemphasisondocumentation. A.TheResearchLogbytheAmericanChemicalSociety. B.Lossoftenureforanalteredresearchlog.

• 1.Doittoday.

2.Checkittomorrowornextweek:italwaysmakessensetoday. 3.Keepupwithdocumentationbytyingittoeventsintheproject. 4.Includefulldatesandnames.

• Arealexample...
• 1.Executioninvolvescarryingouttaskswithineachstep.

2.Effectiveexecutionrequirestherighttools.

• Software

a.Texteditor b.Filemanager c.Statisticalsoftware d.Macroprogram(evenifonlytoinserttimestamps) e.Wordprocessor

• Hardware:display,storage,memory,CPU

3.Planningisprobablymoreimportantthancomputingpower.

• For example…
• Cornell 1975: the entire computing infrastructure
• IBM370with240Kmemory
• Winchesterdriveswith3MBstorage
• Costofcomputing$1,000,000.

Meantimetodegree7.6years.

• Indiana 2009: a disposable PC
• Asus1000HEwith2GBmemory
• FreeAgentwith1TBstorage

10,000timesmore

• 350,000timesmore...
• Costofcomputing$400(2,500timesless).

Meantimetodegree7.6years.

• 1.Randomlydivideyourselvesintotwogroups.
• Thecomputerscancomputewhenevertheywantto.
• Theplannerscanonlycomputefortwosixhoursessionsaweek.

2.Whofinishesfirst?

• Principlesforacomputingworkflow

1.Dualworkflow:keepdatamanagementanddataanalysisseparate. 2.Runorder:namefilessothatiftheyarereruninalphabeticalorder,youwill produceexactlythesameresults. 3.Postingprincipleforsharingresults(definedlater)

• Datamanagement==>
• <==Dataanalysis
• Run order and a dual workflow

Datamanagement

• Dataanalysis

data01.do stat01a.do data02V2.do stat01b.do data03.do stat01cV2.do data03-1.do data03-2.do stat02a.do data04.do stat02a1.do stat02b.do stat03aV2.do stat03b.do stat03c.do stat03c1.do stat03c2V2.do stat03d.do

• postingprinciple

Thepostingprincipleisdefinedbytworules: 1.Thesharerule:Onlyshareresultsafterthefilesareposted. 2.Thenochangerule:Onceafileisposted,neverchangeit.

• 1.Theyareselfcontained

2.Theyincludeversioncontrol(version 11.1) 3.Theyexcludedirectoryinformation(whichmightchange) 4.Theyexplicitlysetseedsforrandomnumbers 5.Theyrequirethatyouarchiveuserwrittenadofiles

• Simplyput:Itshouldrunonanothercomputeratalaterdatewithoutchanges.
• 1.Lotsofthoughtfulcomments

2.Alignment,indentationandspacing 3.Shortlineswithoutwrapping 4.Noambiguousabbreviations: l a l in 1/3

• +----------------+

| Key | |----------------| | frequency | | row percentage | +----------------+ | Years of education Occupation | 3 6 7 8 9 10 11 12 13 | Total

• ----------+----------------------------------------------------------------------------
• ----------------------+----------

Menial | 0 2 0 0 3 1 3 12 2 | 31 | 0.00 6.45 0.00 0.00 9.68 3.23 9.68 38.71 6.45 | 100.00

• ----------+----------------------------------------------------------------------------
• ----------------------+----------

BlueCol | 1 3 1 7 4 6 5 26 7 | 69 | 1.45 4.35 1.45 10.14 5.80 8.70 7.25 37.68 10.14 | 100.00

• ----------+----------------------------------------------------------------------------
• ----------------------+----------

Craft | 0 3 2 3 2 2 7 39 7 | 84 | 0.00 3.57 2.38 3.57 2.38 2.38

• 1.Muchofdataanalysisinvolvesrepetitivetasks.

2.Repetitioninviteserrors. 3.Automationisfaster,andlesserrorprone. A. macros:wordsthatrepresentstringsoftext. B.loops:multipleexecutionofthesamecommands. C.returnedresults:avoidingtypingthevalueofanystatisticalresult. D. matrices:holdandsummarizekeyresults. E.adofiles:writeprogramsthatdowhatyouwant. F.me.hlp:don’tkeeplookingupthesamethings.Forexample,…

• help me
• InStata,type:
• findit snag

snagcollectsdozensorhundredsofresultstomakethemeasiertodigest.

• Thestandardoutputisusedtoverifytheresults.
• The“snagged”summaryletsyoudiscoverwhatyouwant.
• Anyoneusingmarginsknowswhythisisnecessary.
• Example:ownsexandownsexucausedweeksofconfusion.

• Cleaning 1a: finding an error with a graph
• Cleaning 1b: reversing the graph
• Cleaning 2: remembering a coding decision
• Cleaning 3: understanding the substantive process
• Cleaning 4: avoiding expensive mistakes

• 1.Takelotsofclassesinstatistics.

2.Findexemplars;don’trediscoverthewheel;don’tdoit“yourway”.

• 1.Contentandmethodsaresubstantive,disciplinarydecisions.

2.Presentationsandpreservationofprovenanceareuniversal.

• mlogit (N=337): Factor Change in the Odds of occ

• -----------------+---------------------------------------------

• Variable: ed (sd=2.9464271)

• -----------------+---------------------------------------------

• Variable: exper (sd=13.959364)
• ThecircledtextcontainsresultsImayneedtoconfirmlater:
• Turningon"show/hide¶"revealstheprovenance:
• twoway (line art_root2 art_root3 art_root4 art_root5 articles, ///

lwidth(medium)), ytitle(Number of Publications to the k-th Root) /// yscale(range(0 8.)) legend(pos(11) rows(4) ring(0)) /// caption(wf7-caption.do \ jsl 2008-04-09, size(vsmall))

• Whenitcomestosavingyourwork,expectthingstogowrong,expectthatyou

willdeletethewrongfileattheworstpossibletime,andexpectahosetobeleft

• nintheroomaboveyourcomputer.Ifyouexpecttheworst,youmightbeable

topreventit.

• 1.KennedyassassinationonNovember22,1963andthe9/11survey.

2.508KvolumesinobsoleteformatsatBritishMuseum.2MvideosatIU. 3.NeilArmstrong'swalkonthemoononJuly20,1969,thelostmoontapes, andPinkFloyd'sDarkSideoftheMoon.

• "afuzzygrayblobwadingthroughaninkwell" DarkSideoftheMoon

• 1. Installtheprogam
• 2. Dropfilesintothefolder
• 3. RetrievethemfromanymachinewithDropbox
• 4. Havesharedfoldersforcollaboration
• 5. Avoidsendingattachmentsevenforonetimefileexchanges
• Dropboxandsimilarservices,enterprisemassstorage,localservers.
• 1.Sizeperdriveincreasedbyafactorofmorethan300,000.

2.Costpergigabytedecreasedbyafactorof7,000,000. 3.AshoeboxfullofportabledrivescanholdenoughIBMcardstofilla30M cubicfootbuilding;60Mcubicfeetnextmonth.Withcompression…

• 1.Slowly,systematically,throughtfully.

2.Finishthelast5%ofthechange. 3.LikePennandTeller,masterafewcooltricks. 4.Don'tdoitunderdeadline.

• 1.Therearemanyviableworkflows.

2.ThekeyadvantageoftheWFbookisthatitiswrittendown. 3.AlanAcockwrote:

• “Noteveryonewillagreewithallof[Long's]suggestions.”
• “IwillposttheannouncementofWorkflowonmydoorwiththefollowing

note:‘Iamgladtohelpanybodywhofollowedatleast25%oftheadvice Longprovides—andbringsmetheirdofiles!’” 4.DoyoureallywanttospendyourtimerediscoveringthemistakesImade?