JSMeter: Characterizing the Behavior of JavaScript Web Applications - PowerPoint PPT Presentation

JSMeter: Characterizing the Behavior of JavaScript Web Applications Paruj Ratanaworabhan Kasetsart University, Thailand Ben Livshits and Ben Zorn Microsoft Research, Redmond in collaboration with David Simmons, Corneliu Barsan, and Allen Wirfs-Brock 1

Why Measure JavaScript? • Standardized, de facto language for the web – Support in every browser, much existing code • Browser and JavaScript performance is important – Are current JavaScript benchmarks representative? – Limited understanding of JavaScript behavior in real sites • Who cares? – Users, web application developers, JavaScript engine developers 3

ZDNet 29 May 2008 Browser Wars! ghacks.net Dec. 2008 4

Artificial Benchmarks versus Real World Sites 7 V8 8 SunSpider 11 real sites: JSMeter programs : programs: • richards • 3-draytrace • deltablue • access-nbody • crypto • bitops-nsieve • raytrace • controlflow Maps Maps Goals of JSMeter Project • earley- • crypto-aes boyer • Instrument JavaScript execution and measure behavior • date-xparb • regexp • Compare behavior of JavaScript benchmarks against real sites • math-cordic • splay • Consider how benchmarks can mislead design decisions • string-tagcloud 5

How We Measured JavaScript Source-level instrumentation \ie\jscript\*.cpp custom jscript.dll website visits custom trace files 100% 90% Offline 80% 70% 60% 50% analyzers Constant 40% Other Str Ops 30% 20% Concat Op 10% 0% custom trace files 6

Visiting the Real Sites • Getting past page load performance • Attempted to use each site in “normal” way: amazon Search a book, add to shopping cart, sign in, and sign out bing Type in a search query and also look for images and news bingmap Search for a direction from one city to another cnn Read front page news ebay Search for a notebook, bid, sing in, and sign out economist Read front page news, view comments facebook Log in, visit a friend pages, browse through photos and comments gmail Sign in, check inbox, delete a mail, and sign out google Type in a search query and also look for images and news googlemap Search for a direction from one city to another hotmail Sign in, check inbox, delete a mail, and sign out 7

Understanding JavaScript Behavior Code JavaScript Events Objects 8

Code Behavior • Function size • Instructions/call • Code locality Code • Instruction mix JavaScript Events Objects 9

Sour urce ce size ze (kilo lobytes ytes) Total Bytes of JavaScript Source 1000 1500 2000 2500 500 0 amazon bing bingmap cnn Real Sites V8 SunSpider ebay economist facebook gmail google googlemap hotmail richards deltablue crypto raytrace earley regexp splay 3d-raytrace access-nbody bitops-nsieve controlflow crypto-aes date-xparb Code|Objects|Events math-cordic regexp-dna string-tagcloud 10

Static Unique Functions Executed Unique que Funct ctio ions ns Execut uted 1,000 1,500 2,000 2,500 3,000 3,500 4,000 500 0 amazon bing bingmap cnn Real Sites V8 SunSpider ebay economist facebook gmail google googlemap hotmail richards deltablue crypto raytrace earley regexp splay 3d-raytrace access-nbody bitops-nsieve controlflow crypto-aes date-xparb Code|Objects|Events math-cordic regexp-dna string-tagcloud 12

Code|Objects|Events Bytecodes / Call 300.00 function(a,b) { var i=0,elem,pos=a.length; if(D.browser.msie) { while(elem=b[i++]) 250.00 if(elem.nodeType!=8) a[pos++]=elem; } else while(elem=b[i++]) 200.00 ll a[pos++]=elem; Call return a es / C } ecodes 150.00 Byteco 100.00 50.00 0.00 amazon bing bingmap cnn ebay economist facebook gmail google googlemap hotmail richards deltablue crypto raytrace earley regexp splay 3d-raytrace access-nbody bitops-nsieve controlflow crypto-aes date-xparb math-cordic regexp-dna string-tagcloud 15 Real Sites V8 SunSpider

Fract ction ion of Code e Execu cuted ted 100.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% 90.00% 0.00% Fraction of Code Executed amazon bing not executed Most code Real Sites V8 SunSpider bingmap cnn ebay economist facebook gmail google googlemap hotmail richards deltablue crypto raytrace earley regexp splay 3d-raytrace access-nbody bitops-nsieve controlflow crypto-aes date-xparb math-cordic regexp-dna Code|Objects|Events string-tagcloud 16

Object Allocation Behavior • Allocation by types • Live heap composition Code • Lifetime distribution JavaScript Events Objects 17

Total heap data (kilobytes tes) 10000 15000 20000 25000 30000 35000 40000 45000 5000 0 amazon Total Bytes Allocated bing Real Sites V8 SunSpider bingmap cnn ebay economist facebook gmail google googlemap hotmail richards deltablue crypto raytrace earley regexp splay 3d-raytrace access-nbody bitops-nsieve controlflow Code|Objects|Events crypto-aes date-xparb math-cordic regexp-dna string-tagcloud 18

Rest are strings Many functions Heap Data by Type 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 0 1 amazon bing Real Sites V8 SunSpider bingmap cnn Heap Data by Type ebay economist facebook gmail google googlemap hotmail richards deltablue crypto raytrace earley regexp splay V8 aggragate 3d-raytrace access-nbody bitops-nsieve controlflow crypto-aes date-xparb math-cordic Code|Objects|Events allocate much data Few benchmarks regexp-dna string-tagcloud aggregate Script Func Arrays String Native Func Date Error Math RegExp Enumarator Objects 19

Code|Objects|Events Live Heap Over Time (gmail) Functions grow steadily GC reduces size of heap Objects grow steadily too 20

Code|Objects|Events Live Heap over Time (ebay) Heaps Heap contains repeatedly mostly functions created, discarded Heap drops to 0 on page load 21

Code|Objects|Events 2 Search Websites, 2 Architectures You stay on the same page during your entire visit Code loaded once Heap is bigger Every transition loads a new page Code loaded repeatedly Heap is smaller Bing Google 22

Event Handlers in JavaScript • Number of events • Sizes of handlers Code JavaScript Events Objects 26

Code|Objects|Events Event-driven Programming Model • Single-threaded, non-preemptive event handlers • Example handlers: onabort, onclick, etc. • Very different from batch processing of benchmarks • Handler responsiveness critical to user experience 27

Code|Objects|Events Total Events Handled 7,000 6,000 5,000 led ndled s Hand 4,000 nts l Event 3,000 tal Tota 2,000 Almost no events 1,000 0 Real Sites V8 28

Code|Objects|Events Median Bytecodes / Event Handled 2137 506 500 450 ed ndled 400 ent Hand 350 Event 300 des / E 250 ecodes n Byteco 200 150 dian Medi 100 50 0 30

Sure, this is all good, but… • Everyone knows benchmarks are unrepresentative • How much difference does it make, anyway? • Wouldn’t any benchmarks have similar issues? 33

Cold-code Experiment • Observation – Real web apps have lots of code (much of it cold) – Benchmarks do not • Question: What happens if the benchmarks have more code? – We added extra, unused to code to 7 SunSpider benchmarks – We measured the impact on the benchmark performance 34

Performance Impact of Cold Code 900 900 800 800 700 700 600 600 Time (msec) Time (msec) 0K 0K 500 500 200K 200K 400 400 400K 400K 300 800K 300 800K 1M 1M 200 200 2M 2M 100 100 0 0 Chrome IE 8 3.0.195.38 8.0.601.18865 Cold code makes Cold code has SunSpider on Chrome non-uniform impact up to 4.5x slower on execution time 35

Impact of Benchmarks • What gets emphasis – Making tight loops fast – Optimizing small amounts of code • Important issues ignored – Garbage collection (especially of strings) – Managing large amounts of code – Optimizing event handling – Considering JavaScript context between page loads 36

Conclusions • JSMeter is an instrumentation framework – Used to measure and compare JavaScript applications – High-level views of behavior promote understanding • Benchmarks differ significantly from real sites – Misleads designers, skews implementations • Next steps – Develop and promote better benchmarks – Design and evaluate better JavaScript runtimes – Promote better performance tools for JavaScript developers 37

Additional Resources • Project : http://research.microsoft.com/en-us/projects/jsmeter/ • Video : Project JSMeter: JavaScript Performance Analysis in the Real World" - MSDN Channel 9 interview with Erik Meier, Ben Livshits, and Ben Zorn • Paper: – “ JSMeter: Comparing the Behavior of JavaScript Benchmarks with Real Web Applications”, Paruj Ratanaworabhan, Benjamin Livshits and Benjamin G. Zorn, USENIX 2010 Conference on Web Application Development (WebApps’10), June 2010. • Additional measurements: – "JSMeter: Characterizing Real-World Behavior of JavaScript Programs", Paruj Ratanaworabhan, Benjamin Livshits, David Simmons, and Benjamin Zorn, MSR-TR-2009-173, December 2009 (49 pages), November 2009. 38

Additional Slides 39