Learning Networking by Reproducing Results Lisa Yan, Lecturer in - - PowerPoint PPT Presentation

▶
learning networking by reproducing results
SMART_READER_LITE
LIVE PREVIEW

Learning Networking by Reproducing Results Lisa Yan, Lecturer in - - PowerPoint PPT Presentation

May 27, 2023 481 likes •1.11k views

Learning Networking by Reproducing Results Lisa Yan, Lecturer in Computer Science Stanford University June 24, 2020 Slides http://stanford.edu/~yanlisa/publications/precs20_yanlisa-slides.pdf 1 Suppose you need to teach 1. Introduction to

slide-1
SLIDE 1

Learning Networking by Reproducing Results

Lisa Yan, Lecturer in Computer Science Stanford University June 24, 2020

1

Slides http://stanford.edu/~yanlisa/publications/precs20_yanlisa-slides.pdf

slide-2
SLIDE 2

Lisa Yan, 2020

2

Suppose you need to teach…

  • 1. Introduction to Networking
  • 2. Graduate Networking
slide-3
SLIDE 3

Lisa Yan, 2020

Introduction to networking

3

Application Transport Network Link

slide-4
SLIDE 4

Lisa Yan, 2020

Graduate networking

4

slide-5
SLIDE 5

Lisa Yan, 2020

Graduate networking

5

Train and build experience in order to become a future networking researcher or networking engineer.

slide-6
SLIDE 6

Lisa Yan, 2020

What kinds of systems should advanced students build?

6

Give them all the same project

(a bit boring) (too risky)

Have them create their own project

slide-7
SLIDE 7

Lisa Yan, 2020

What kinds of systems should advanced students build?

7

Assignment goals

  • build a system
  • think critically about a system

?

circa 2012: the beginning of Mininet, a realistic network emulator

slide-8
SLIDE 8

Lisa Yan, 2020

What kinds of systems should advanced students build?

8

Assignment goals

  • build a system
  • think critically about a system

Lisa Y Yan and Nick McKeown. Learning Networking by Reproducing Research

  • Results. CCR April 2017. Best o
  • f C

CCR a award a at S SIG IGCOMM 2 2017.

? Reproduce someone else’s research.

circa 2012: the beginning of Mininet, a realistic network emulator

slide-9
SLIDE 9

Lisa Yan, 2020

9

How was our experience?

Lisa Y Yan and Nick McKeown. Learning Networking by Reproducing Research

  • Results. CCR April 2017. Best o
  • f C

CCR a award a at S SIG IGCOMM 2 2017.

slide-10
SLIDE 10

Lisa Yan, 2020

10

Really, really cool.

Lisa Y Yan and Nick McKeown. Learning Networking by Reproducing Research

  • Results. CCR April 2017. Best o
  • f C

CCR a award a at S SIG IGCOMM 2 2017.

slide-11
SLIDE 11

Lisa Yan, 2020

11

These projects…

  • 1. Spark discussions between researchers and students.
  • 2. Give students more tools to use in their own research.
  • 3. Jumpstart careers in networking.

Provide a fully reproducible project in the public domain.

Lisa Y Yan and Nick McKeown. Learning Networking by Reproducing Research

  • Results. CCR April 2017. Best o
  • f C

CCR a award a at S SIG IGCOMM 2 2017.

slide-12
SLIDE 12

Lisa Yan, 2020

Today

Reproducing research project: Graduate computer networks

  • Project overview
  • Student stories

Greater impacts

  • A stronger research community
  • A framework for education

12

slide-13
SLIDE 13

Lisa Yan, 2020

Today

Reproducing research project: Graduate computer networks

  • Project overview
  • Student stories

Greater impacts

  • A stronger research community
  • A framework for education

13

slide-14
SLIDE 14

Lisa Yan, 2020

CS 244 Reproducibility Project

14

  • 1. Project p

proposal

  • Pick a paper and a key result to reproduce.
  • Contact the original researchers

Day 1 7 14 21 28

  • 2. In

Intermediate r report

  • Preliminary work
  • TA-student meeting for next steps
  • 4. Peer d

discussion

In-class presentations

  • 3. Final r

report

  • Blog post
  • Public source code
  • Steps for reproducing

reproducingnetworkresearch.wordpress.com

slide-15
SLIDE 15

Lisa Yan, 2020

Research venues

15

top networking conferences

internet standards

systems security

slide-16
SLIDE 16

Lisa Yan, 2020

Experiment details: Original research

16

B4 Wide Area Network (WAN) Facebook 2000- node cluster NetFPGA, Programmable solutions Theoretical models

ns-2

network traffic simulators and emulators

Mininet - http://mininet.org/ Mahimahi - http://mahimahi.mit.edu/

world wide web

slide-17
SLIDE 17

Lisa Yan, 2020

17

How can we reproduce research with limited resources?

slide-18
SLIDE 18

Lisa Yan, 2020

18

  • 1. Use simulators and emulators

where necessary.

slide-19
SLIDE 19

Lisa Yan, 2020

  • 1. Use simulators and emulators where necessary.

19

B4 Wide Area Network (WAN) Facebook 2000- node cluster NetFPGA, Programmable solutions Theoretical models

ns-2 Mininet - http://mininet.org/ Mahimahi - http://mahimahi.mit.edu/

world wide web network traffic simulators and emulators

slide-20
SLIDE 20

Lisa Yan, 2020

20

  • 2. Use cloud computing resources.
slide-21
SLIDE 21

Lisa Yan, 2020

  • 2. Use cloud computing resources.

21

slide-22
SLIDE 22

Lisa Yan, 2020

22

  • 3. Ask the original authors!
slide-23
SLIDE 23

Lisa Yan, 2020

  • 3. Ask the original authors!

23

System source code Workload generation Open-source 33% Open-source but

  • ut-of-date/inconsistent

18% Part of Linux Kernel 10% Contacted a author 7% Binary available 1% Student-created 19% Not-needed 12% Open-source 19% Sufficient details in paper 40% Student-created 41%

slide-24
SLIDE 24

Lisa Yan, 2020

24

What have our students achieved?

slide-25
SLIDE 25

Lisa Yan, 2020

Research topics

25

Spark

TCP

video streaming

73 unique published papers, 1993–2018

slide-26
SLIDE 26

Lisa Yan, 2020

0% 20% 40% 60% 80% 100% 2019 2018 2017 2016 2015 2014 2013 2012

% of student groups Course offering

Unsuccessful Successful

9 years of student projects

26

73 unique papers reproduced 300+ students since 2012 (150+ projects)

2018: require new research reproductions 2016: introduced mahimahi emulator

slide-27
SLIDE 27

Lisa Yan, 2020

Reproduced work, by popularity

27

2 4 6 8 10 # student reproductions Unique papers (73 total)

Pre-2018 Post-2018

  • 1. An Argument for

Increasing TC TCP’s Initial Congestion Window (2010)

  • 2. Jellyfish: Networking D

Data Ce Centers Randomly (2012) 3.

  • 3. TC

TCP Fast Open (2011)

  • 4. Confused, timid, and

unstable: picking a vi vide deo streaming r rate is hard (2014)

(require new unique research reproductions)

slide-28
SLIDE 28

Lisa Yan, 2020

28

In these projects, our students learn a lot about engineering networked systems.

slide-29
SLIDE 29

Lisa Yan, 2020

Quick refresher: Congestion control

Netw twork c congesti tion: overloading network link, preventing useful communication TCP congestion avoidance:

  • 1. Increase sending window

slowly (additively) with receiver acknowledgments (ACKs)

  • 2. If data loss, decrease sending

window quickly (multiplicatively)

29

# packets sent time 1. 2.

data

Sender Receiver

ack

slide-30
SLIDE 30

Lisa Yan, 2020

TCP opt-ack attack

30

Original result from paper

  • R. Sherwood et al. Misbehaving TCP receivers can cause internet-wide

congestion collapse. CCS 2005.

Optimistic acknowledgments (opt-acks) encourage victim senders to send more

slide-31
SLIDE 31

Lisa Yan, 2020

Reproduced: TCP opt-ack attack

31

Original result from paper Students’ reproduced result (2016, blog post)

  • R. Sherwood et al. Misbehaving TCP receivers can cause internet-wide

congestion collapse. CCS 2005.

ns-2 (simulator) Mininet (emulator)

slide-32
SLIDE 32

Lisa Yan, 2020

Choosing a video streaming rate

32

Original result from paper (2012)

T.-Y. Huang et al. Confused, Timid, and Unstable: Picking a Video Streaming Rate is Hard. IMC 2012.

Overly conservative video streaming rates lead to dismally low throughput

slide-33
SLIDE 33

Lisa Yan, 2020

Reproduced: Choosing a video streaming rate (2013)

33

Original result from paper (2012)

T.-Y. Huang et al. Confused, Timid, and Unstable: Picking a Video Streaming Rate is Hard. IMC 2012.

Students’ reproduced result (2013, blog post)

Our experiments use the real backend servers of [Netflix]. We do not use Mininet or any other form of network emulation.

slide-34
SLIDE 34

Lisa Yan, 2020

Reproduced: Choosing a video streaming rate (2013)

34

Original result from paper (2012)

T.-Y. Huang et al. Confused, Timid, and Unstable: Picking a Video Streaming Rate is Hard. IMC 2012.

Students’ reproduced result (2013, blog post)

We can easily observe that the problem reported in the paper has since been fixed in [Netflix]…We h have contacted t the p paper au authors and they confirm that this is the case.

slide-35
SLIDE 35

Lisa Yan, 2020

Reproduced: Choosing a video streaming rate (2017)

35

Original result from paper (2012)

T.-Y. Huang et al. Confused, Timid, and Unstable: Picking a Video Streaming Rate is Hard. IMC 2012.

Students’ reproduced result (20 2017, blog post)

We chose to start with examining [Vimeo and YouTube], since they are freely a accessible without a subscription and there exist third party tools … for manipulating video downloads…

slide-36
SLIDE 36

Lisa Yan, 2020

Replicated: Choosing a video streaming rate (2017)

36

Original result from paper (2012)

T.-Y. Huang et al. Confused, Timid, and Unstable: Picking a Video Streaming Rate is Hard. IMC 2012.

Students’ reproduced result (2017, blog post)

Experimental results show that Yo YouTube’s player does not exhibit the downward spiral effect…

slide-37
SLIDE 37

Lisa Yan, 2020

Replicated: Choosing a video streaming rate (2017)

37

Original result from paper (2012)

T.-Y. Huang et al. Confused, Timid, and Unstable: Picking a Video Streaming Rate is Hard. IMC 2012.

Students’ reproduced result (2017, blog post)

…while Vi Vimeo’s player does.

slide-38
SLIDE 38

Lisa Yan, 2020

AWStream

38

Original result from paper GitHub open-source code Adaptive streaming in wide-area networks (geo-distributed sites, scarce/variable bandwidth)

  • B. Zhang et al. AWStream: Adaptive Wide-Area Streaming Analytics. SIGCOMM 2018.
slide-39
SLIDE 39

Lisa Yan, 2020

Reproduced: AWStream

39

Original result from paper Students’ reproduced result (2019, blog post)

  • B. Zhang et al. AWStream: Adaptive Wide-Area Streaming Analytics. SIGCOMM 2018.
slide-40
SLIDE 40

Lisa Yan, 2020

Reproduced: AWStream

40

Original result from paper

  • B. Zhang et al. AWStream: Adaptive Wide-Area Streaming Analytics. SIGCOMM 2018.

Students’ reproduced result (2019, blog post)

…overall we found [the documentation] to be outdated and at times misleading. We relied mostly on close reading of the code and email c correspondence with t the o

  • riginal

paper’s ’s a authors for guidance.

slide-41
SLIDE 41

Lisa Yan, 2020

41

What about unsuccessful research reproductions?

slide-42
SLIDE 42

Lisa Yan, 2020

Overambitious engineering Emulator performance restrictions

42

Common scenarios

“We spent our last week trying to find a mixed LP optimizer.” (reproduction of FastMPC, SIGCOMM 2015, blog post)

Differences in workloads

“Average QoE measurements were much higher than those reported…our Wifi/International Links more than capable

  • f delivering high quality video streams”

(reproduction of Pensieve, SIGCOMM 2017, blog post) "We scaled down all load generation parameters, but we still couldn’t achieve target latencies when emulating on a single machine.” (reproduction of QJump, NSDI 2015, blog post)

slide-43
SLIDE 43

Lisa Yan, 2020

43

How does time affect research artifacts?

(at least) two examples:

  • 1. Linux kernel versions
  • 2. Web traffic
slide-44
SLIDE 44

Lisa Yan, 2020

Linux kernel versions: PRR for TCP

PRR: Proportional Rate Reduction paces out retransmissions across received ACKs

44

  • N. Dukkipati, et al. Proportional Rate Reduction for TCP. IMC 2011.

The two (additional) discrepancies in /net/ipv4/tcp_input.c do not have a big impact on the experiment results. (2015, blog post)

365 commits to /net/ipv4/tcp_input.c

Ubuntu version Linux kernel 2011 11.10 3.0 No PRR 2014 12.04 3.2 With PRR

12 commits reference PRR

Late 2016 17.04+ 4.10+ Option to turn

  • ff PRR

Present-day 20.04+ 5.4+

slide-45
SLIDE 45

Lisa Yan, 2020

HTTPS and web traffic

In 2014, HTTPS page load times were slower than HTTP.

45

Original result from paper (Alexa top 500 websites)

  • D. Naylor et al. The Cost of the “S” in HTTPS. CoNEXT 2014.
  • 40% of sites: HTTPS

noticeably slower

  • 55% of sites: no difference
  • <3% of sites: HTTPS faster
slide-46
SLIDE 46

Lisa Yan, 2020

HTTPS and web traffic

In 2014, HTTPS page load times were slower than HTTP.

46

Original result from paper (Alexa top 500 websites)

  • D. Naylor et al. The Cost of the “S” in HTTPS. CoNEXT 2014.

Students’ reproduced result (2017, blog post)

(over 4G) (over Fiber)

  • 15% of sites: HTTPS

noticeably slower (vs 40%)

  • 80%: no difference (vs 55%)
  • 30%: HTTPS faster (vs <3%)
slide-47
SLIDE 47

Lisa Yan, 2020

TCP Fast Open

Page Load Time (PLT) much higher in recent years Emulator: Dummynet

47

Pag Page RT RTT (ms ms) PLT: : no non-TF TFO O (s (s) PLT: T : TFO (s (s) Im Improve- me ment: CoNEXT 2011 Amazon 100 2.60 2.34 10% NYTimes 100 4.59 4.30 6% Students 2015 Amazon 100 15.92 12.55 21% NYTimes 100 5.37 4.03 25% (2015, blog post)

  • S. Radhakrishnan et al. TCP Fast Open. CoNEXT 2011.
slide-48
SLIDE 48

Lisa Yan, 2020

48

(pause)

slide-49
SLIDE 49

Lisa Yan, 2020

49

So what?

slide-50
SLIDE 50

Lisa Yan, 2020

Today

Reproducing research project: Graduate computer networks

  • Project overview
  • Student stories

Greater impacts

  • A stronger research community
  • A framework for education

50

slide-51
SLIDE 51

Lisa Yan, 2020

A stronger research community

51

Student Original researcher New researcher Simulator/ emulator developer

slide-52
SLIDE 52

Lisa Yan, 2020

…overall we found [the documentation] to be outdated and at times misleading. We relied mostly on close reading of the code and email c correspondence with t the o

  • riginal

paper’s ’s a authors for guidance.

  • B. Zhang et al. AWStream: Adaptive Wide-Area Streaming Analytics. SIGCOMM 2018.

AWStream (SIGCOMM 2018, Students 2019)

52

Original result from paper Students’ reproduced result (2019, blog post)

slide-53
SLIDE 53

Lisa Yan, 2020

QJump (NSDI 2015, Students 2015)

“Their assumption was that [people] would reproduce the results in an actual d datacenter, whereas we did the emulation i in Mi Mininet.” “In the end, we did not use their scripts directly, but it was nice to see that the authors were enthusiastic to have their work reproduced.”

53

  • M. Grosvenor et al. Queues don’t matter when you can JUMP them! NSDI 2015.

(2015, blog post)

slide-54
SLIDE 54

Lisa Yan, 2020

A stronger research community

54

Student Original researcher New researcher Simulator/ emulator developer I learned how to implement a scheduler for my graduate research! I can confirm and improve my current and past research!

slide-55
SLIDE 55

Lisa Yan, 2020

A stronger research community

55

Student Original researcher New researcher Simulator/ emulator developer I just started a career in networks, and this prepared me for the real world.

Tool feedback/ development

Mininet

http://mininet.org/

Mahimahi

http://mahimahi.mit.edu/

slide-56
SLIDE 56

Lisa Yan, 2020

A stronger research community

56

Student Original researcher New researcher Simulator/ emulator developer

A fully reproducible project in the public domain.

  • Other researchers can

build upon it

  • Eases technology

transfer

We were contacted by both the original authors and a student working on his own research!

slide-57
SLIDE 57

Lisa Yan, 2020

57

How can we go beyond networks?

slide-58
SLIDE 58

Lisa Yan, 2020

Example assignment schedule (10-week)

58

Assignment 1 1 Core topic practice emulation environment

Week 1 mid-quarter 10

Assignment 2 2 Core topic practice emulation environment Final p project Reproduce research

slide-59
SLIDE 59

Lisa Yan, 2020

Example assignment schedule (10-week)

59

Assignment 1 1 Core topic practice emulation environment

Week 1 mid-quarter 10

Assignment 2 2 Reproduce the same project Final p project Reproduce research,

  • r original work
slide-60
SLIDE 60

Lisa Yan, 2020

For platform developers

60

Assignment 1 1 Core topic practice emulation environment

Week 1 mid-quarter 10

Assignment 2 2 Core topic practice emulation environment Final p project Reproduce research

  • Provide a list of papers/suggested projects

to get students started

  • Be accessible and responsive to a

multitude of applications

slide-61
SLIDE 61

Lisa Yan, 2020

Encouraging community-led reproducible research

61

This talk Reproducible research for all

slide-62
SLIDE 62

Thank you!

cs244.stanford.edu/reproducibility

62

  • L. Yan and N. McKeown. Learning Networking by Reproducing Research Results. CCR April 2017.

https://ccronline.sigcomm.org/2017/learning-networking-by-reproducing-research-results/

Nick McKeown, Keith Winstein, Sachin Katti, Bruce Spang CS 244: Advanced Topics in Networking