Evaluation and Reproducibility of Numerical Simulations Stephan - - PowerPoint PPT Presentation

▶
evaluation and reproducibility of numerical simulations
SMART_READER_LITE
LIVE PREVIEW

Evaluation and Reproducibility of Numerical Simulations Stephan - - PowerPoint PPT Presentation

Aug 17, 2022 104 likes •618 views

W ESTFLISCHE W ILHELMS -U NIVERSITT M NSTER Evaluation and Reproducibility of Numerical Simulations Stephan Rave living knowledge Software Tool Seminar Mnster WWU Mnster WS14/15 October 15, 2014 W ESTFLISCHE W ILHELMS -U

slide-1
SLIDE 1

Evaluation and Reproducibility of Numerical Simulations

Stephan Rave

Software Tool Seminar WS14/15 Münster October 15, 2014

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

slide-2
SLIDE 2

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 2 /14

Outline

◮ Greenhorn Mistakes... ◮ Creation and Management of Simulation Datasets for the MULTIBAT project ◮ Reproducibility Stephan Rave (stephan.rave@wwu.de)

slide-3
SLIDE 3

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 3 /14

Greenhorn Mistakes

Stephan Rave (stephan.rave@wwu.de)

slide-4
SLIDE 4

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 3 /14

Greenhorn Mistakes

◮ Write super cool program. Stephan Rave (stephan.rave@wwu.de)

slide-5
SLIDE 5

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 3 /14

Greenhorn Mistakes

◮ Write super cool program. ◮ Hmm, will take a few days. Stephan Rave (stephan.rave@wwu.de)

slide-6
SLIDE 6

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 3 /14

Greenhorn Mistakes

◮ Write super cool program. ◮ Hmm, will take a few days. ◮ Lock laptop in office over the weekend. Stephan Rave (stephan.rave@wwu.de)

slide-7
SLIDE 7

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 3 /14

Greenhorn Mistakes

◮ Write super cool program. ◮ Hmm, will take a few days. ◮ Lock laptop in office over the weekend. ◮ Weekend! :) Stephan Rave (stephan.rave@wwu.de)

slide-8
SLIDE 8

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 3 /14

Greenhorn Mistakes

◮ Write super cool program. ◮ Hmm, will take a few days. ◮ Lock laptop in office over the weekend. ◮ Weekend! :) ◮ Monday morning: Error! Calculation terminated! :( Stephan Rave (stephan.rave@wwu.de)

slide-9
SLIDE 9

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 4 /14

Greenhorn Mistakes II

Stephan Rave (stephan.rave@wwu.de)

slide-10
SLIDE 10

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 4 /14

Greenhorn Mistakes II

◮ Write super cool program. Stephan Rave (stephan.rave@wwu.de)

slide-11
SLIDE 11

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 4 /14

Greenhorn Mistakes II

◮ Write super cool program. ◮ Let’s test it first on a small grid ... Stephan Rave (stephan.rave@wwu.de)

slide-12
SLIDE 12

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 4 /14

Greenhorn Mistakes II

◮ Write super cool program. ◮ Let’s test it first on a small grid ... ◮ Start giant computation on office machine. Stephan Rave (stephan.rave@wwu.de)

slide-13
SLIDE 13

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 4 /14

Greenhorn Mistakes II

◮ Write super cool program. ◮ Let’s test it first on a small grid ... ◮ Start giant computation on office machine. ◮ Weekend! :) Stephan Rave (stephan.rave@wwu.de)

slide-14
SLIDE 14

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 4 /14

Greenhorn Mistakes II

◮ Write super cool program. ◮ Let’s test it first on a small grid ... ◮ Start giant computation on office machine. ◮ Weekend! :) ◮ Monday morning: Updates installed, calculation terminated! :( Stephan Rave (stephan.rave@wwu.de)

slide-15
SLIDE 15

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 5 /14

Intermezzo: Screen

Stephan Rave (stephan.rave@wwu.de)

slide-16
SLIDE 16

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 6 /14

Greenhorn Mistakes III

Stephan Rave (stephan.rave@wwu.de)

slide-17
SLIDE 17

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 6 /14

Greenhorn Mistakes III

◮ Write super cool program and test it. Stephan Rave (stephan.rave@wwu.de)

slide-18
SLIDE 18

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 6 /14

Greenhorn Mistakes III

◮ Write super cool program and test it. ◮ Start giant computation on BIGDADDY or METATRON. Stephan Rave (stephan.rave@wwu.de)

slide-19
SLIDE 19

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 6 /14

Greenhorn Mistakes III

◮ Write super cool program and test it. ◮ Start giant computation on BIGDADDY or METATRON. ◮ Weekend! :) (Each hour, nervously test with screen if everything is still ok.) Stephan Rave (stephan.rave@wwu.de)

slide-20
SLIDE 20

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 6 /14

Greenhorn Mistakes III

◮ Write super cool program and test it. ◮ Start giant computation on BIGDADDY or METATRON. ◮ Weekend! :) (Each hour, nervously test with screen if everything is still ok.) ◮ Monday morning: Cool, error really goes down! Stephan Rave (stephan.rave@wwu.de)

slide-21
SLIDE 21

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 6 /14

Greenhorn Mistakes III

◮ Write super cool program and test it. ◮ Start giant computation on BIGDADDY or METATRON. ◮ Weekend! :) (Each hour, nervously test with screen if everything is still ok.) ◮ Monday morning: Cool, error really goes down! ◮ Advisor: What about the L2-norm? Stephan Rave (stephan.rave@wwu.de)

slide-22
SLIDE 22

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 6 /14

Greenhorn Mistakes III

◮ Write super cool program and test it. ◮ Start giant computation on BIGDADDY or METATRON. ◮ Weekend! :) (Each hour, nervously test with screen if everything is still ok.) ◮ Monday morning: Cool, error really goes down! ◮ Advisor: What about the L2-norm? ◮ Days later: Yes, also in L2-norm! Stephan Rave (stephan.rave@wwu.de)

slide-23
SLIDE 23

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 6 /14

Greenhorn Mistakes III

◮ Write super cool program and test it. ◮ Start giant computation on BIGDADDY or METATRON. ◮ Weekend! :) (Each hour, nervously test with screen if everything is still ok.) ◮ Monday morning: Cool, error really goes down! ◮ Advisor: What about the L2-norm? ◮ Days later: Yes, also in L2-norm! ◮ Advisor: The plot would look much nicer if it was green! Stephan Rave (stephan.rave@wwu.de)

slide-24
SLIDE 24

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 6 /14

Greenhorn Mistakes III

◮ Write super cool program and test it. ◮ Start giant computation on BIGDADDY or METATRON. ◮ Weekend! :) (Each hour, nervously test with screen if everything is still ok.) ◮ Monday morning: Cool, error really goes down! ◮ Advisor: What about the L2-norm? ◮ Days later: Yes, also in L2-norm! ◮ Advisor: The plot would look much nicer if it was green! ◮ After deadline: Green plot. Stephan Rave (stephan.rave@wwu.de)

slide-25
SLIDE 25

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 7 /14

The Punchline

Stephan Rave (stephan.rave@wwu.de)

slide-26
SLIDE 26

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 7 /14

The Punchline

Separate simulation from evaluation and visualization!

Stephan Rave (stephan.rave@wwu.de)

slide-27
SLIDE 27

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 7 /14

The Punchline

Separate simulation from evaluation and visualization!

◮ Generate as much data as you can! Stephan Rave (stephan.rave@wwu.de)

slide-28
SLIDE 28

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 7 /14

The Punchline

Separate simulation from evaluation and visualization!

◮ Generate as much data as you can! ◮ But keep in mind what you want to know! Stephan Rave (stephan.rave@wwu.de)

slide-29
SLIDE 29

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 7 /14

The Punchline

Separate simulation from evaluation and visualization!

◮ Generate as much data as you can! ◮ But keep in mind what you want to know! ◮ Some questions will only arise later on! Stephan Rave (stephan.rave@wwu.de)

slide-30
SLIDE 30

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 8 /14

Difficulties

Stephan Rave (stephan.rave@wwu.de)

slide-31
SLIDE 31

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 8 /14

Difficulties

◮ Which format? Stephan Rave (stephan.rave@wwu.de)

slide-32
SLIDE 32

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 8 /14

Difficulties

◮ Which format? ◮ With which parameters have I created this data31.dat? (metadata) Stephan Rave (stephan.rave@wwu.de)

slide-33
SLIDE 33

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 8 /14

Difficulties

◮ Which format? ◮ With which parameters have I created this data31.dat? (metadata) ◮ After a year: What the heck was important-simulation2.dat all about?

(more metadata)

Stephan Rave (stephan.rave@wwu.de)

slide-34
SLIDE 34

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 8 /14

Difficulties

◮ Which format? ◮ With which parameters have I created this data31.dat? (metadata) ◮ After a year: What the heck was important-simulation2.dat all about?

(more metadata)

◮ ls data* | wc -l: 297. In which file did I have p = 3? (searchability) Stephan Rave (stephan.rave@wwu.de)

slide-35
SLIDE 35

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 8 /14

Difficulties

◮ Which format? ◮ With which parameters have I created this data31.dat? (metadata) ◮ After a year: What the heck was important-simulation2.dat all about?

(more metadata)

◮ ls data* | wc -l: 297. In which file did I have p = 3? (searchability) ◮ Before submission: Did I already have this bug when I made the

simulation?!?!?! (reproducibility)

Stephan Rave (stephan.rave@wwu.de)

slide-36
SLIDE 36

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 9 /14

The Solution

1http://neuralensemble.org/sumatra

Stephan Rave (stephan.rave@wwu.de)

slide-37
SLIDE 37

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 9 /14

The Solution

◮ Ignore all existing software and roll your own!

1http://neuralensemble.org/sumatra

Stephan Rave (stephan.rave@wwu.de)

slide-38
SLIDE 38

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 9 /14

The Solution

◮ Ignore all existing software and roll your own! ◮ In particular, ignore Sumatra1!

1http://neuralensemble.org/sumatra

Stephan Rave (stephan.rave@wwu.de)

slide-39
SLIDE 39

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 10 /14

Empirical Operator Interpolation

◮ Input: ◮ problem ◮ operator ◮ parameter training set ⊂ R2 ◮ maximum amount of interpolation points ◮ number of parameters for testing ◮ Output: ◮ interpolation points, interpolation basis ◮ interpolation error for each training parameter at each time for each

number of interpolation points

◮ interpolation error for each test parameter at each time for each number

  • f interpolation points

Stephan Rave (stephan.rave@wwu.de)

slide-40
SLIDE 40

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 11 /14

Demo

Stephan Rave (stephan.rave@wwu.de)

slide-41
SLIDE 41

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 12 /14

Reproducibility

Stephan Rave (stephan.rave@wwu.de)

slide-42
SLIDE 42

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 12 /14

Reproducibility

◮ Put all code under version control! Stephan Rave (stephan.rave@wwu.de)

slide-43
SLIDE 43

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 12 /14

Reproducibility

◮ Put all code under version control! ◮ Archive only datasets which have been created with a clean working tree. Stephan Rave (stephan.rave@wwu.de)

slide-44
SLIDE 44

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 12 /14

Reproducibility

◮ Put all code under version control! ◮ Archive only datasets which have been created with a clean working tree. ◮ Save commit hash along with dataset. Stephan Rave (stephan.rave@wwu.de)

slide-45
SLIDE 45

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 12 /14

Reproducibility

◮ Put all code under version control! ◮ Archive only datasets which have been created with a clean working tree. ◮ Save commit hash along with dataset. ◮ Save the script/program invocation which has created the dataset. Stephan Rave (stephan.rave@wwu.de)

slide-46
SLIDE 46

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 12 /14

Reproducibility

◮ Put all code under version control! ◮ Archive only datasets which have been created with a clean working tree. ◮ Save commit hash along with dataset. ◮ Save the script/program invocation which has created the dataset. ◮ Write a reproduce script, which auto-generates the dataset. Stephan Rave (stephan.rave@wwu.de)

slide-47
SLIDE 47

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 13 /14

More Reproducibility

2https://www.virtualbox.org/ 3https://www.docker.com/

Stephan Rave (stephan.rave@wwu.de)

slide-48
SLIDE 48

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 13 /14

More Reproducibility

◮ When creation visualizations, specify uniquely which datasets shall be used.

2https://www.virtualbox.org/ 3https://www.docker.com/

Stephan Rave (stephan.rave@wwu.de)

slide-49
SLIDE 49

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 13 /14

More Reproducibility

◮ When creation visualizations, specify uniquely which datasets shall be used. ◮ Write a reproduce script, which automatically generates all datasets for the

given visualization.

2https://www.virtualbox.org/ 3https://www.docker.com/

Stephan Rave (stephan.rave@wwu.de)

slide-50
SLIDE 50

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 13 /14

More Reproducibility

◮ When creation visualizations, specify uniquely which datasets shall be used. ◮ Write a reproduce script, which automatically generates all datasets for the

given visualization.

◮ Use virtual machines with the same initial state for all your computations.

(VirtualBox2, docker3 or a combination of both).

2https://www.virtualbox.org/ 3https://www.docker.com/

Stephan Rave (stephan.rave@wwu.de)

slide-51
SLIDE 51

living knowledge WWU Münster

WESTFÄLISCHE WILHELMS-UNIVERSITÄT MÜNSTER

Evaluation and Reproducibility of Numerical Simulations 14 /14

That’s it!

Stephan Rave (stephan.rave@wwu.de)