A Very Short History Feldman & Sutherland (1979) Rejuvenating Experimental Computer Science ACM Executive Committee position on the crisis in experimental computer science (1979) Viewpoints to Starting from the 1980s, hundreds of arguments on experimental computer science Experimental CS Tichy (1998) Should Computer Scientists Matti Tedre, Stockholm University Experiment More? matti.nospam.tedre@acm.nospam.org 1 2 Outline 1.More 1. Should Computer 2.Computer scientists Scientists Experiment 3.Experiment More? 4.Should How much do we 5.Challenges for actually experiment? Experimental CS 3 4
How Much Do We How Much Do We Experiment? (1/2) Experiment? (2/2) According to one (old) study: 40% of articles about new designs and models lack experimentation 50% of SE articles TICHY ET AL., CACM 23(10):544 GLASS ET AL., CACM 47(6):91 5 6 Where Should We Pioneers in Seek for Experiments? Sorting + Searching Where’s the experimentation? Are we really justified to say “all work in computing needs experimentation”? Need to look closer to what people mean by “computer science” DENNING, CACM 23(10):544 KNUTH, TAOCP, VOL.3 7 8
Is Computing a Science? 2. Should Computer Dijkstra (1987), McKee (1995), Brooks (1996), Hoare No! (early works) Scientists Experiment Denning (2007), Simon (1969), Newell et al. (1969), More? Yes. Rosenbloom (2004), Ralston & Shaw (1980), Minsky (1979), McCracken (1979), ... What do we mean by “computer science” here? Yes, but... Knuth (2001), Tichy (1998), Hartmanis (1993/1994), Vessey et al. (2002), Fletcher (1995) 9 10 ...If So, What Is It a How About the Science of? Computing Fields? Computers Hamming (1969) Computers + related phenomena Newell et al. (1969) Algorithms + related phenomena Knuth (1974) Information processing Forsythe (1967) Complexity Simon (1969) Classes of computations Dijkstra (1972) Programming Khalil & Levy (1978) Information processes & transf. Denning et al. (1981) ACM/IEEE CURRICULA (2005) 11 12
How Do Computer Scientists Work? Mathematical analysis Project monitoring 3. Should Computer Conceptual analysis Case study What is the role of Concept implementation Assertion Surveys (quant/qual) Field study experiments in Scientists Experiment Screening Replicated experiment Effects analysis Synthetic experiment computer science Benchmarking Forecasting More? methodology? Simulation Game/role playing Emulation Interviews Participatory design Focus groups What do computer Hermeneutics Content analysis scientists mean by Dynamic analysis Ethnography “experiment”? Simulation Grounded theory Legacy data analysis Critical theory Lessons learned Incremental development Static analysis Performance analysis 13 14 5 Views to Experiments 1. Demonstration in Computer Science Experiment 1.Demonstration experiment It’s not known if task t can be automated 2.Trial experiment efficiently / reliably / feasibly / cost-efficiently etc. A demonstration of experimental technology 3.Field experiment “EXPERIMENT”?!?! shows that it can be done 4.Comparison experiment ...”experiment”? 5.Controlled experiment 15 16
2. Trial Experiment 3. Field Experiment It’s not known how well system p meets its specifications / performs It’s not known how well a system works in its sociotechnical context A trial evaluation is designed to experiment (“test”) with the qualities of the product Common in information systems (Palvia et al., 2003) Can be lab-based or in the use environment Offers more control than case studies and surveys; typically a quasi-experiment or limited-control E.g., performance analysis (w/o comparison) experiment Benchmarking (Gustedt et al., 2009) 17 18 4. Comparison 5. Controlled Experiment Experiment It is not known if algorithm A outperforms B with data set d and parameters p It is not known if x and y are associated, or if x An experiment is set up to measure and compare causes y . A( d , p ) and B( d , p ) The gold standard of scientific work Typical of incremental development work where Enables generalization and prediction the aim is to do task x better Typically doesn’t follow the blinding principle 19 20
Argumentum ad How do we justify saying “We Antiquitatem SHOULD experiment more”? One of the most famous arguments for increasing 4. Should Computer experimentation in computing (Tichy et al., 1995) Scientists Experiment More? TICHY ET AL., J. SYST. SOFTW. 28:9–18 21 22 2) How Unique is 1) How About Physics? Computing journals Computer Science? Am. J. of Physics Maybe computing is just different? Compare*... Conclusion: we The work that leads to Nobel Prize in Physics should experiment less ? The work that leads to Turing Awards If it’s a unique field, does it not have unique ways of working, too? ZELKOWITZ & WALLACE, INF. & SW TECH. 39:735–743 *HARTMANIS, PROC. 13TH CONF. OFOST: 1–12 23 24
Challenge 1: Sloppy terminology 5. Challenges for Experimental CS But computing has always been liberal with terms! ZELKOWITZ & WALLACE, IEEE COMPUTER 31(5):23–31 25 26 Challenge 2: Empirical Challenge 3: Develop vs. Experimental Experimental Protocols When we say “experimental”, do we really mean “empirical”? In many branches of CS there are standards for data, parameters, and measurement! FLETCHER, J. SYS. SOFTW. 30(1995):161–163 27 28
Challenge 4: Tacked-on Challenge 5: “Hypotheses” “Computer Science” When we say “CS” do we really mean “SE”, or “Health informatics” or “educational technology”? CS projects can be great even if they aren’t based on hypothesis testing! DENNING, CACM 23(10):544 29 30 Challenge 6: Arguments Challenge 7: Language to Other Disciplines from the 1600s Is it really necessary to compare with physics? In our arguments, why are we still referring to Why not astronomy? ideas about science developed in the 1600s? ...mathematics? We nowadays know much better how science works. ...economics? 31 32
Thanks! Questions, comments? Obvious errors? matti.nospam.tedre@acm.nospam.org 33
Recommend
More recommend
