Learning Networking by Reproducing Results Lisa Yan, Lecturer in Computer Science Stanford University June 24, 2020 Slides http://stanford.edu/~yanlisa/publications/precs20_yanlisa-slides.pdf 1
Suppose you need to teach… 1. Introduction to Networking 2. Graduate Networking 2 Lisa Yan, 2020
Introduction to networking Application Transport Network Link 3 Lisa Yan, 2020
Graduate networking 4 Lisa Yan, 2020
Graduate networking Train and build experience in order to become a future networking researcher or networking engineer. 5 Lisa Yan, 2020
What kinds of systems should advanced students build? Give them all the (a bit boring) same project Have them create (too risky) their own project 6 Lisa Yan, 2020
What kinds of systems should advanced students build? Assignment goals ? • build a system • think critically about a system circa 2012: the beginning of Mininet, a realistic network emulator 7 Lisa Yan, 2020
What kinds of systems should advanced students build? Reproduce Assignment goals someone else’s ? • build a system research. • think critically about a system circa 2012: the beginning of Mininet, a realistic network emulator Lisa Y Yan and Nick McKeown. Learning Networking by Reproducing Research 8 Lisa Yan, 2020 Results. CCR April 2017. Best o of C CCR a award a at S SIG IGCOMM 2 2017 .
How was our experience? Lisa Y Yan and Nick McKeown. Learning Networking by Reproducing Research 9 Lisa Yan, 2020 Results. CCR April 2017. Best o of C CCR a award a at S SIG IGCOMM 2 2017 .
Really, really cool. Lisa Y Yan and Nick McKeown. Learning Networking by Reproducing Research 10 Lisa Yan, 2020 Results. CCR April 2017. Best o of C CCR a award a at S SIG IGCOMM 2 2017 .
These projects… 1. Spark discussions between researchers and students. 2. Give students more tools to use in their own research. 3. Jumpstart careers in networking. Provide a fully reproducible project in the public domain. Lisa Y Yan and Nick McKeown. Learning Networking by Reproducing Research 11 Lisa Yan, 2020 Results. CCR April 2017. Best o of C CCR a award a at S SIG IGCOMM 2 2017 .
Today Reproducing research project: Graduate computer networks • Project overview • Student stories Greater impacts • A stronger research community • A framework for education 12 Lisa Yan, 2020
Today Reproducing research project: Graduate computer networks • Project overview • Student stories Greater impacts • A stronger research community • A framework for education 13 Lisa Yan, 2020
CS 244 Reproducibility Project 2. In Intermediate r report 4. Peer d discussion • Preliminary work • TA-student meeting for next steps In-class presentations Day 1 7 14 21 28 1. Project p proposal 3. Final r report • Pick a paper and a key result to reproduce. • Blog post • Contact the original researchers • Public source code • Steps for reproducing reproducingnetworkresearch.wordpress.com 14 Lisa Yan, 2020
Research venues top networking conferences systems internet standards security 15 Lisa Yan, 2020
Experiment details: Original research NetFPGA, B4 Wide Area Programmable Network (WAN) solutions Facebook 2000- Theoretical node cluster models world wide web ns-2 network traffic Mininet - http://mininet.org/ simulators and Mahimahi - http://mahimahi.mit.edu/ emulators 16 Lisa Yan, 2020
How can we reproduce research with limited resources? 17 Lisa Yan, 2020
1. Use simulators and emulators where necessary. 18 Lisa Yan, 2020
1. Use simulators and emulators where necessary. NetFPGA, B4 Wide Area Programmable Network (WAN) solutions Facebook 2000- Theoretical node cluster models world wide web ns-2 network traffic Mininet - http://mininet.org/ simulators and Mahimahi - http://mahimahi.mit.edu/ emulators 19 Lisa Yan, 2020
2. Use cloud computing resources. 20 Lisa Yan, 2020
2. Use cloud computing resources. 21 Lisa Yan, 2020
3. Ask the original authors! 22 Lisa Yan, 2020
3. Ask the original authors! System source code Workload generation Open-source 33% Open-source 19% Open-source but 18% Sufficient details in 40% out-of-date/inconsistent paper Part of Linux Kernel 10% Student-created 41% Contacted a author 7% Binary available 1% Student-created 19% Not-needed 12% 23 Lisa Yan, 2020
What have our students achieved? 24 Lisa Yan, 2020
Research topics video streaming TCP Spark 73 unique published papers, 1993–2018 25 Lisa Yan, 2020
9 years of student projects 73 unique papers reproduced 300+ students since 2012 (150+ projects) 2012 2013 Course offering 2014 2015 2016: introduced 2016 mahimahi emulator 2017 2018: require new 2018 research reproductions 2019 0% 20% 40% 60% 80% 100% % of student groups Unsuccessful Successful 26 Lisa Yan, 2020
Reproduced work, by popularity 1. An Argument for 10 Increasing TC TCP ’s Initial Pre-2018 Congestion Window Post-2018 8 # student reproductions (2010) (require new unique research 2. Jellyfish: Networking D Data 6 reproductions) Centers Randomly (2012) Ce 4 TCP Fast Open (2011) 3. 3. TC 4. Confused, timid, and 2 unstable: picking a vi vide deo 0 rate is hard streaming r Unique papers (73 total) (2014) 27 Lisa Yan, 2020
In these projects, our students learn a lot about engineering networked systems. 28 Lisa Yan, 2020
Quick refresher: Congestion control Netw twork c congesti tion : overloading data network link, preventing useful communication ack Sender Receiver TCP congestion avoidance: 1. Increase sending window slowly (additively) with receiver # packets sent acknowledgments (ACKs) 1. 2. If data loss, decrease sending window quickly (multiplicatively) 2. time 29 Lisa Yan, 2020
TCP opt-ack attack Optimistic acknowledgments (opt-acks) encourage victim senders to send more Original result from paper R. Sherwood et al. Misbehaving TCP receivers can cause internet-wide congestion collapse. CCS 2005. 30 Lisa Yan, 2020
Reproduced: TCP opt-ack attack ns-2 (simulator) Mininet (emulator) Students’ reproduced result Original result from paper (2016, blog post) R. Sherwood et al. Misbehaving TCP receivers can cause internet-wide congestion collapse. CCS 2005. 31 Lisa Yan, 2020
Choosing a video streaming rate Overly conservative video streaming rates lead to dismally low throughput Original result from paper (2012) T.-Y. Huang et al. Confused, Timid, and Unstable: Picking a Video Streaming Rate is Hard. IMC 2012. 32 Lisa Yan, 2020
Reproduced: Choosing a video streaming rate (2013) Our experiments use the real backend servers of [Netflix]. We do not use Mininet or any other form of network emulation. Original result from paper (2012) Students’ reproduced result (2013, blog post) T.-Y. Huang et al. Confused, Timid, and Unstable: Picking a Video Streaming Rate is Hard. IMC 2012. 33 Lisa Yan, 2020
Reproduced: Choosing a video streaming rate (2013) We can easily observe that the problem reported in the paper has since been fixed in [Netflix]… We h have contacted t the p paper authors and they confirm au that this is the case. Original result from paper (2012) Students’ reproduced result (2013, blog post) T.-Y. Huang et al. Confused, Timid, and Unstable: Picking a Video Streaming Rate is Hard. IMC 2012. 34 Lisa Yan, 2020
Reproduced: Choosing a video streaming rate (2017) We chose to start with examining [Vimeo and YouTube], since they are freely a accessible without a subscription and there exist third party tools … for manipulating video downloads… Original result from paper (2012) Students’ reproduced result ( 20 2017 , blog post) T.-Y. Huang et al. Confused, Timid, and Unstable: Picking a Video Streaming Rate is Hard. IMC 2012. 35 Lisa Yan, 2020
Replicated : Choosing a video streaming rate (2017) Experimental results show that Yo YouTube ’s player does not exhibit the downward spiral effect… Original result from paper (2012) Students’ reproduced result (2017, blog post) T.-Y. Huang et al. Confused, Timid, and Unstable: Picking a Video Streaming Rate is Hard. IMC 2012. 36 Lisa Yan, 2020
Replicated : Choosing a video streaming rate (2017) …while Vi Vimeo ’s player does. Original result from paper (2012) Students’ reproduced result (2017, blog post) T.-Y. Huang et al. Confused, Timid, and Unstable: Picking a Video Streaming Rate is Hard. IMC 2012. 37 Lisa Yan, 2020
AWStream Adaptive streaming in wide-area networks (geo-distributed sites, scarce/variable bandwidth) Original result from paper GitHub open-source code B. Zhang et al. AWStream: Adaptive Wide-Area Streaming Analytics. SIGCOMM 2018. 38 Lisa Yan, 2020
Reproduced: AWStream Original result from paper Students’ reproduced result (2019, blog post) B. Zhang et al. AWStream: Adaptive Wide-Area Streaming Analytics. SIGCOMM 2018. 39 Lisa Yan, 2020
Recommend
More recommend
