Timelapse Interactive record/replay for web apps Richard J. Brian - PowerPoint PPT Presentation

Timelapse Interactive record/replay for web apps Richard J. Brian Burg Andrew J. Ko Michael D. Ernst Bailey Computer Science and Engineering University of Washington 1

“It’s hard to debug failures in the field” Distributed across time and space Hardware and software variation Users are not software testers 2

Users can’t report failures accurately Works for me.. “a piece doesn’t rotate properly!!” end-user encounters files bug report with developer unable to bug in production ad-hoc information reproduce the bug code 3

Users can’t report failures accurately Works for me.. “a piece doesn’t rotate properly!!” “The most severe problems were errors in steps to reproduce and incomplete information .” 4 “What makes a good bug report”. Zimmerman et al. TSE Vol. 36, No 5 2010

Users can’t report failures accurately Works for me.. “a piece doesn’t rotate properly!!” Bug reporters and developers want better tool support for reproducing buggy behavior. 5 “What makes a good bug report”. Zimmerman et al. TSE Vol. 36, No 5 2010

Existing tools are imprecise and hard to use macro replay Capture and simulate user input. Designed for test and task (CoScripter, Selenium, automation. Sikuli) Selenium/ CoScripter Sikuli Script WebDriver Leshed et al, CHI 2008 Yeh et al, UIST 2009 6

Existing tools are imprecise and hard to use macro replay Capture and simulate user input. Designed for test and task (CoScripter, Selenium, automation. Sikuli) Nondeterministic. Requires extra setup ahead of time. Can’t use with a debugger. deterministic Save and reuse nondeterministic inputs replay to exactly recreate a specific (Mugshot, WaRR) Play/pause buttons only. Slows execution. down execution. Can’t use with a debugger. 7

Timelapse: a precise, fast, integrated replay tool This talk: • An interface for capturing and replaying program behavior • Techniques for cheap, precise record/replay in web browsers • How developers use record/replay during debugging tasks 8

How to capture program behavior 9

How to navigate a recording 10

Using replay while debugging 11

Browsers interpret input, render output Browser Input Web Interpreter Output (WebKit, Gecko) (User, Network, Timers) 12

Timelapse captures a browser’s inputs Inputs Log Browser Input Web Interpreter Output (WebKit, Gecko) (User, Network, Timers) 13

Timelapse replays a browser’s inputs Inputs Log Web Interpreter Output (WebKit, Gecko) 14

Browsers have layered architectures User input, Date.now , commands win.colorDepth event loop policy decisions callbacks Embedders Web Interpreter Platforms (Firefox, Safari, (WebKit, Gecko) Chrome) 15

Timelapse intercepts input at layer boundaries Embedders Web Interpreter Platforms (Firefox, Safari, (WebKit, Gecko) Chrome) 16

Inspired by VM record/replay VM record/replay Browser record/replay Hardware interrupts Async callbacks • • Nondeterministic Nondeterministic APIs • • instructions DOM event counts • Instruction counts • 17

Memoizing nondeterministic APIs During normal execution, Date.now() returns the current time. /* file: Source/wtf/DateMath.h */ inline double jsCurrentTime() oo(a,b) { return a + b; } { (c) { return “now:”+ Date.now(); } return floor(WTF::currentTimeMS()); } Platform API Web Interpreter 18

Memoizing nondeterministic APIs During recording, the return value of Date.now() is saved. /* file: Source/wtf/DateMath.h */ inline double jsCurrentTime() oo(a,b) { return a + b; } { (c) { return “now:”+ Date.now(); } return floor(WTF::currentTimeMS()); } Inputs Log Platform API Web Interpreter 19

Memoizing nondeterministic APIs On replay, the logged return value of Date.now() is used. /* file: Source/wtf/DateMath.h */ inline double jsCurrentTime() oo(a,b) { return a + b; } { (c) { return “now:”+ Date.now(); } return floor(WTF::currentTimeMS()); } Inputs Log Platform API Web Interpreter

Making callbacks deterministic enqueue() Callback Event Loop registered while (true) { var event = queue.pop(); this.dispatchToListeners(event); } timerFired() Problem: accurately capturing and simulating event loop Callback dispatches. executes

Making callbacks deterministic enqueue() Callback Event Loop registered while (true) { var event = queue.pop(); this.dispatch(event); } timerFired() Callback timer 42, executes Inputs Log 34 DOM events

Making callbacks deterministic enqueue() Callback Event Loop registered while (true) { var event = queue.pop(); this.dispatchToListeners(event); ... } ... ... timerFired() 34 DOM events! Callback Inputs Log executes

Runtime overheads are acceptable 1.4 1.2 1 Baseline Run times Recording 0.8 (multiple of baseline) 1× Replay 0.6 Seeking 0.4 0.2 0 24

Recordings are small and compressible 800 recording resources log In-memory Site duration on page growth 700 Serialized (s) (KB) (KB/sec) 600 Compresse JSLinux 10.5 4500 0.8 d 500 JS Raytracer 6.3 5.9 1.6 400 Space 300 25.8 712 2.2 Invaders 200 Mozilla.org 22.3 2800 1.3 100 CodeMirror 16.6 168 1.0 0 JSLinux JS Raytracer Space Invaders Mozilla.org CodeMirror Colorpicker DuckDuckGo Colorpicker 15.3 577 1.7 Size (KB) DuckDuckGo 14.1 1900 2.1 25

Page resources dominate recording size 5000 recording resources log In-memory Site duration on page growth 4500 Serialized (s) (KB) (KB/sec) 4000 Compresse 3500 JSLinux 10.5 4500 0.8 d 3000 JS Raytracer 6.3 5.9 1.6 2500 Space 2000 25.8 712 2.2 Invaders 1500 Mozilla.org 22.3 2800 1.3 1000 500 CodeMirror 16.6 168 1.0 0 JSLinux JS Raytracer Space Invaders Mozilla.org CodeMirror Colorpicker DuckDuckGo Colorpicker 15.3 577 1.7 Size (KB) DuckDuckGo 14.1 1900 2.1 26

How would developers use it? 20+ developers with industry Study Design experience within-subjects, 2 tasks per person, 45 minutes per task, 4 treatments RQ: changes to Reproduction frequency/duration? RQ: complete tasks more Performance quickly? more successfully? Who, why? 27

How did developers use it? 20+ developers with industry Study Design experience within-subjects, 2 tasks per person, 45 minutes per task, 4 treatments Shorter and more frequent repro Reproduction actions; Time spent unchanged ( max. 25%; Successful developers quickly avg. 15%) integrated replay into their existing Performance workflows. Unsuccessful developers who used opportunistic strategies were distracted. 28

Current & Future Work Interaction histories aid navigation, Visualizations but not program understanding. Precision and low overhead don’t Passive matter if you forget to start capturing. capturing Developers can gather more runtime Post-hoc data without reproducing behavior: analysis Post-hoc logging, Post-hoc Whyline, Post-hoc SeeSS, Testcase extraction

Conclusion Record/Replay Virtual machine replay techniques work well when applied to web applications. Interaction histories supported–but Visualizations didn’t reduce–reproduction of program state. Replay infrastructure enables new Infrastructure research, tools and workflows. github.com/burg/timelapse

Replay fidelity and completeness Divergence detection supports piecewise implementation. Web interpreters expose a large and ever-changing API. Timelapse doesn’t tame all sources of nondeterminism (yet). Excepting untamed sources, the DOM tree and JavaScript heap are identical for all recorded and replayed executions. Possible divergence is automatically detected when: • DOM event counts differ on capture and replay • Memoized inputs are overused or unused • Network request details differ unexpectedly • Known-bad APIs are used by client code 31

Interpreter inputs by source User: mouse, keyboard, scroll, resize Network : images, scripts, HTML, AJAX Commands : page navigation Internal nondeterminism: Animations, transitions, multimedia, async script and parser yields Functions : Date.now, Math.random, etc Caching : resources, cookies Timers : timer schedule 32

Shim: the thing in the middle Shims are used to implement deterministic record/replay. The hard part of implementing record/replay is designing and placing shims. 33

Embedding and platform APIs Abstraction layers separate web interpreters from platforms/embedders. PLATFORM API EMBEDDING API Embedder Web Interpreter Platforms s (WebKit, Gecko) 34

Embedding and platform APIs Abstraction layers separate web interpreters from platforms/embedders. PLATFORM API EMBEDDING API Embedder Web Interpreter Platforms s (WebKit, Gecko) 35

Embedding and platform APIs Shims sit between the web interpreter and abstraction layers. PLATFORM API EMBEDDING API Embedder Web Interpreter Platforms s (WebKit, Gecko) 36

Embedding and platform APIs Shims sit between the web interpreter and abstraction layers. PLATFORM API EMBEDDING API Embedder Web Interpreter Platforms s (WebKit, Gecko) 37