The yt Project: Growing & Engaging a Community of Practice - - PowerPoint PPT Presentation

▶
the yt project
SMART_READER_LITE
LIVE PREVIEW

The yt Project: Growing & Engaging a Community of Practice - - PowerPoint PPT Presentation

Jul 13, 2023 176 likes •1.26k views

The yt Project: Growing & Engaging a Community of Practice Matthew Turk Columbia University What is yt? astro-ph/1011.3514 astro-ph/1112.4482 yt-project.org yt is a platform for analysis and visualization of astrophysical* simulations

slide-1
SLIDE 1

The yt Project:

Growing & Engaging a Community of Practice

Matthew Turk Columbia University

slide-2
SLIDE 2

What is yt?

slide-3
SLIDE 3

astro-ph/1011.3514 astro-ph/1112.4482 yt-project.org

slide-4
SLIDE 4

yt is a platform for analysis and visualization of astrophysical* simulations

slide-5
SLIDE 5
slide-6
SLIDE 6
slide-7
SLIDE 7

install script: Full dependency stack Source code Development environment GUI Sample data

slide-8
SLIDE 8

There are many simulation codes.

slide-9
SLIDE 9

There are many simulation codes, but there’s only one sky.

slide-10
SLIDE 10

data structures, methods, assumptions, IO formats, units, variable names, ...

slide-11
SLIDE 11
slide-12
SLIDE 12
slide-13
SLIDE 13
slide-14
SLIDE 14

Fully-Supported Semi-Supported In-Progress Enzo FLASH Orion Nyx Uniform Data Chombo Athena ART RAMSES GDF Piernik Cactus Hydra Gadget

slide-15
SLIDE 15

yt is designed to address physical, not computational, entities and questions.

slide-16
SLIDE 16

Orthogonal Rays Non-orthogonal Rays 1D Slices Oblique Slices Projections 2D Spheres Rectangular Prisms Disks/Cylinders Inclined Boxes Clumps Extracted Regions Ellipsoids 3D

slide-17
SLIDE 17

Scripting interface, HTML5 GUI, in-situ processing, transparent multi-level parallelism...

slide-18
SLIDE 18

yt is supposed to get out of the way.

slide-19
SLIDE 19

from yt.mods import * pf = load(’galaxy0030/galaxy0030’) p = SlicePlot(pf, 2, ’Density’, ’c’, (200,’kpc’)) p.save(’Galaxy’)

slide-20
SLIDE 20
slide-21
SLIDE 21
slide-22
SLIDE 22
slide-23
SLIDE 23
slide-24
SLIDE 24

10-12 g/cc

slide-25
SLIDE 25

6 C.B. Hummels et al.

slide-26
SLIDE 26

Canned Analysis Tasks

slide-27
SLIDE 27

Absorption Spectrum Coordinate Transformations Halo Finding Mass Functions Merger Trees Halo Profiling Level Sets Light Cones Light Rays Time Series Star Analysis Two-Point Analysis

slide-28
SLIDE 28

“Community”?

slide-29
SLIDE 29

“Users” “Developers” Traditional View

slide-30
SLIDE 30

“Users” “Developers” Most Scientific Software

slide-31
SLIDE 31

“Devusers” Community of Practice

slide-32
SLIDE 32

“Developers”

slide-33
SLIDE 33

“Developers” Inspection and verification

slide-34
SLIDE 34

“Developers” Inspection and verification Tracking modifications

slide-35
SLIDE 35

“Developers” Inspection and verification Tracking modifications Sharing information

slide-36
SLIDE 36

“Developers” Inspection and verification Tracking modifications Sharing information Adding functionality

slide-37
SLIDE 37

“Developers” Inspection and verification Tracking modifications Sharing information Adding functionality All are necessary characteristics of the scientific process as a whole.

slide-38
SLIDE 38

“Users”

slide-39
SLIDE 39

“Users” Uncritical acceptance of code...?

slide-40
SLIDE 40

“Users” Uncritical acceptance of code...? “These are people we give the code to that don’t care how it works.”

slide-41
SLIDE 41

Developing as a Team

slide-42
SLIDE 42

Challenges

slide-43
SLIDE 43

Reward Structure

slide-44
SLIDE 44

de facto and de jure

slide-45
SLIDE 45

de facto and de jure

  • Utilization of developed tools
  • Respect from community
  • Project involvements
  • Invitations to speak
slide-46
SLIDE 46

de facto and de jure

  • Funding
  • Publications
  • Citation count
  • Influence
slide-47
SLIDE 47

Traditional astrophysics does not favor tool builders.

slide-48
SLIDE 48

Chores

slide-49
SLIDE 49

Chores

Documentation, testing,

  • utreach,

infrastructure.

slide-50
SLIDE 50

Tasks not fully-aligned with reward structure goals present great motivational challenges.

slide-51
SLIDE 51

Co-opetition

slide-52
SLIDE 52

Co-opetition

  • Funding
  • Publications
  • Citation count
  • Influence

( )

slide-53
SLIDE 53

How developer community engagement -- as seen in cohesion, excitement, energy -- is affected by funded improvements is unclear.

slide-54
SLIDE 54

Strategies

slide-55
SLIDE 55

The Art of Community by Bacon Producing Open Source Software by Fogel Team Geek by Fitzpatrick & Collins-Sussman Organizing Simulation Code Collectives by Sundberg Scientific Software Production by Howison & Herbsleb

slide-56
SLIDE 56

You must design the community you want.

slide-57
SLIDE 57

Technical & Social

slide-58
SLIDE 58

Technical & Social

slide-59
SLIDE 59

SVN hg

Repository Users

Users & Repos Users & Repos Users & Repos Users & Repos Users & Repos Users & Repos
slide-60
SLIDE 60

Forky development: very low barrier to entry; everything comes in the box.

slide-61
SLIDE 61

Testing: answer as well as integration tests get run every 30 minutes.

slide-62
SLIDE 62

Code review: eyes on (nearly) every changeset mentoring new developers

slide-63
SLIDE 63

The path upstream must be obvious.

slide-64
SLIDE 64
  • Happy application
  • Itch-scratching
  • Submission
  • Code review & mentoring
  • Participation
slide-65
SLIDE 65

Non-core contributed code: a place for sharing scripts, data and images

slide-66
SLIDE 66
slide-67
SLIDE 67

Communication

slide-68
SLIDE 68

Communication All business is conducted openly.

slide-69
SLIDE 69

Communication Immediate

slide-70
SLIDE 70

2

Communication Low-Latency

slide-71
SLIDE 71

Communication High-Latency

slide-72
SLIDE 72

Technical & Social

slide-73
SLIDE 73

H R T

slide-74
SLIDE 74

Humility R T

slide-75
SLIDE 75

Humility Respect T

slide-76
SLIDE 76

Humility Respect Trust

slide-77
SLIDE 77

Humility Respect Trust

(Fitzpatrick & Collins-Sussman)

slide-78
SLIDE 78

The most important thing I try to remember is that I want to foster a community of peers, not of elites.

slide-79
SLIDE 79
slide-80
SLIDE 80

I’ve noticed something is acting strangely with ... “ “

slide-81
SLIDE 81

I’ve noticed something is acting strangely with ... You’re probably doing it wrong. “ “ “ “

slide-82
SLIDE 82

Can you tell us how you would expect it to act? I’ve noticed something is acting strangely with ... “ “ “ “

slide-83
SLIDE 83

I think that there might be a bug in the way ... “ “

slide-84
SLIDE 84

It’s like that for very good reasons. Don’t touch it. I think that there might be a bug in the way ... “ “ “ “

slide-85
SLIDE 85

It behaves the way it does because ... I think that there might be a bug in the way ... “ “ “ “

slide-86
SLIDE 86

Thoughtfulness, inquiry, confidence, letting go.

slide-87
SLIDE 87

This culture must be seeded directly.

slide-88
SLIDE 88

This culture must be seeded directly, because culture will self-propagate.

slide-89
SLIDE 89

letting go

slide-90
SLIDE 90

Pride rather than ownership

slide-91
SLIDE 91

Projects passing between people

slide-92
SLIDE 92

Too much control can smother.

slide-93
SLIDE 93

Too much control can smother. will

slide-94
SLIDE 94

Successes

slide-95
SLIDE 95
slide-96
SLIDE 96

Developed by working astrophysicists.

slide-97
SLIDE 97
slide-98
SLIDE 98
slide-99
SLIDE 99

Tom Abel David Collins Brian Crosby Andrew Cunningham Nathan Goldbaum Cameron Hummels Ji-hoon Kim Steffen Klemer Kacper Kowalik Michael Kuhlen Eve Lee Chris Malone Chris Moody Andrew Myers Jeff Oishi Jean-Claude Passy Thomas Robitaille Anna Rosen Anthony Scopatz Devin Silvia Sam Skillman Stephen Skory Britton Smith Geoffrey So Casey Stark Elizabeth Tasker Matthew Turk Rick Wagner John Wise John ZuHone

slide-100
SLIDE 100

Tom Abel David Collins Brian Crosby Andrew Cunningham Nathan Goldbaum Cameron Hummels Ji-hoon Kim Steffen Klemer Kacper Kowalik Michael Kuhlen Eve Lee Chris Malone Chris Moody Andrew Myers Jeff Oishi Jean-Claude Passy Thomas Robitaille Anna Rosen Anthony Scopatz Devin Silvia Sam Skillman Stephen Skory Britton Smith Geoffrey So Casey Stark Elizabeth Tasker Matthew Turk Rick Wagner John Wise John ZuHone

slide-101
SLIDE 101

“Usage of a HPC Data Analysis and Visualization System,” Szczepanski et al 2012

Usage on NSF NICS Nautlius

slide-102
SLIDE 102
slide-103
SLIDE 103
slide-104
SLIDE 104

(play movie)

slide-105
SLIDE 105

“... it seems likely that significant software contributions to existing scientific software projects are not likely to be rewarded through the traditional reputation economy of science. Together these factors provide a reason to expect the over-production of independent scientific software packages, and the under- production of collaborative projects in which later academics build on the work of earlier

  • nes.”

Howison & Herbsleb (2011)

slide-106
SLIDE 106

Thank you.

slide-107
SLIDE 107

Tom Abel David Collins Brian Crosby Andrew Cunningham Nathan Goldbaum Cameron Hummels Ji-hoon Kim Steffen Klemer Kacper Kowalik Michael Kuhlen Eve Lee Chris Malone Chris Moody Andrew Myers Jeff Oishi Jean-Claude Passy Thomas Robitaille Anna Rosen Anthony Scopatz Devin Silvia Sam Skillman Stephen Skory Britton Smith Geoffrey So Casey Stark Elizabeth Tasker Matthew Turk Rick Wagner John Wise John ZuHone