Optimized for change: Architecture @ Etsy Kellan Elliott-McCrea - - PowerPoint PPT Presentation

Optimized for change: Architecture @ Etsy

Kellan Elliott-McCrea @kellan CTO, Etsy

Launched June 18, 2005 875,000 active sellers 33.5MM items for sale $65.9MM in sales, in May 1.4B page views, in May 102 engineers 32 releases, last Friday

LAMP

any questions?

Why?

3 inevitabilities we design for:

• 1. Things break, unexpectedly
• 2. What we're building changes
• 3. We don't get to start over

2 years of change.

* Don't bet against the future. * Our customers are humans. * Simplicity always wins, in the end. * Favor global vs local optimization. * Ambiguity kills momentum. * Make failure cheap. * Technical debt is an inevitable by-product

• f shipping code.

* Optimize for change.

Architectural Principles

Cleverness

Complex systems and change

• 1. Distributed systems are inherently complex.
• 2. The outcome of change in complex systems is hard to

predict.

• 3. The outcome of small, frequent, measurable changes

are easier to predict, easier to recover from, and promote learning.

Continuous deployment, Metrics Driven Development, Blameless Post-Mortems

Continuous deployment: Small, frequent changes to production

Continuous Deployment:

No branching.

“All existing revision control systems were built by people who build installed software”

• Paul Hammond,

Always Ship Trunk, Velocity 2010

if ($cfg[‘awesome_new_search’]) { # new hotness $rsp = do_solr(); } else { # boring old stuff $rsp = do_grep(); }

Continuous Deployment:

feature flags

Continuous Deployment:

Ramp - ups

(on top of feature flags)

• 1. Launch to staff only
• 2. Launch to 1% of all users
• 3. Launch to members of a beta group

Continuous Deployment:

any engineer can launch a feature to

1% of users

Continuous Deployment:

~200 experiments live right now

Metrics driven development:

introspection isn’t

• ptional.

measure everything, log everything

Metrics driven development:

Metrics happen when you make it easy. And visible.

Metrics driven development:

holtWintersConfidence(Upper|Lower) Teach computer to read graphs

Metrics driven development:

More info: http://www.slideshare.net/ mikebrittain/metricsdriven-engineering

Optimize for MTTR, not MTBF

How?

Etsy

Etsy

EMR/S3 PCI BCP, Cold

inbound request

etsy.com/ api.etsy.com /atlas etsystatic.com/ photos bcn.etsy.com CDNs - diversified at the DNS level Internet providers - diversified at borders

Etsy

network appliances

AWS

analytics imstor apache php application MySQL search memcache async http StatsD sqlite gearman logs server/OS hardware Squid apache php imstor NFS apache logs logrotate HDFS analytics EMR JRuby/ Cascading S3 PHP MySQL S3 search Thrift Jetty Solr slaves datasets Solr master HBase sharded MySQL MySQL dbindex dbshards dbaux dbdata mail out SMTP X-Yarnblaster etc

PCI

via jsonp, no privileged access

CDNs: Put a slider on it Just works via weighted DNS

Apache

* Well known * PHP is native * apache_note * fast start time * cheap in place replacement * .htaccess * Challenge: memory usage

Apache: apache_note

apache_note('etsy_uaid', $id); A d d i t i v e ! i n s a n e l y u s e f u l ! i n t r

• s

p e c t i

• n

t h r

• u

g h t h e l i f e c y c l e

LogFormat "%{X-Forwarded-For}i % {True-Client-IP}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User- Agent}i\" % {etsy_shop_id}n % {etsy_uaid}n %V % {etsy_ab_selections}n % {etsy_request_uuid}n % {etsy_api_consumer_key}n % {etsy_api_method_name}n % {php_memory_usage_bytes}n % {php_time_microsec}n %D" combined

Apache: log format

Etsy: the App

* 487,000 lines of PHP * 214,000 lines of Javascript * Monolithic codebase * 3 front ends, Etsy.com, API, Atlas

Etsy: the App

* routing handled by Apache * scripts fronting OO PHP5 * PHP, fast by default * opcode caching * Challenge: liveliness when calling services

Etsy: coding patterns

* light weight, home rolled “framework” * ORM handles DAO across backends * config and feature flags systems used everywhere * small slow moving datasets stored as PHP arrays * A/B tests * Smarty * StatsD * Concurrency * memcache

Etsy: A/B tests

* beaconed * inserted into logs via apache_note * conditionalized on feature flags * nightly reports on conversion, bounce rate, etc * nightly reports on page speed, memory usage, etc

Etsy: Smarty

* pre-compiled * pre-compiled per language

Etsy: StatsD StatsD::increment("logins.success"); StatsD::timing("gearman.time", $msec);

* 340,000 application metrics

Etsy: Concurrency

* no native concurrency in PHP * asynchronous HTTP calls * Gearman

Etsy: Async HTTP calls

* curl_multi_exec * non-blocking, per request time outs * used for optional aspects of a page * curl against http://localhost to avoid network overhead

Etsy: Gearman

* language agnostic job server * don’t use an MQ when you want a job server * 150 job types * persistent jobs flushed to MySQL, read from memory * non-persistent jobs just stored in memory * NP queue is wicked fast.

Etsy: Gearman

* scaling CPU of cron jobs * denormalizing data * pushing to 3rd party services

Etsy: Challenges

* Apache memory usage * liveliness talking to services, no concurrency, blocking by default

Etsy: graph of distributed failure

Etsy: Challenges

* Apache memory usage * liveliness talking to services: no concurrency, blocking by default

Enforce liveliness with a judicious application of force

Etsy: judicious application of force

list($v, $res, $shar) = @fopen(‘/proc/self/statm', 'r'); $mine = $res-$shar; if ($mine > $cfg[‘sizelimit’]) { $pid = getmypid(); @exec("kill -USR1 $pid"); }

Etsy: judicious application of force

Bowhunter * Find long running PHP processes * Try to avoid those mid-post

• pen(APACHE, "/usr/bin/curl -s http://localhost/server-

status|") || die "$!";

Etsy: judicious application of force

Query_killer * Same idea, long running queries * MySQL “SHOW PROCESSLIST();”

Memcache

* Caching, obviously * Cache invalidation is hard * Write buffering * multi_get * rate limits

Memcache

* atomic INCR is awesome * slice your time windows to reduce risk of cache eviction * we’ve been unlucky, lots of segfaults :( * multi_get slows down the more boxes in the pool

MySQL: By the numbers * 25K+queries/sec avg * 3TB InnoDB buffer pool * 15TB + data stored * 50 servers * 99.99% queries under 1ms

MySQL: a NotMuchSQL server * no joins * no foreign keys * no transactions or locks * no sub-selects * store data like you want to read it. * also: no auto_increment

MySQL: a NotMuchSQL server “Normalization is for sissie.”

• Cal Henderson, Flickr

MySQL: scale horizontally * objects shared by key * lookups maintained in dbindex (MySQL is a FAST key-value store) * avoid key hashing, range partitions, and partitioning functions

more: http://www.slideshare.net/jgoulah/the-etsy-shard-architecture-starts-with-s-and-ends-with-hard

MySQL: Master-Master * objects hashed to a side, avoid split brain * allows in place schema upgrades without slave promotion * simplified capacity planning

more: http://codeascraft.etsy.com/2012/04/20/two-sides-for-salvation/

web0038 : [Mon Jun 18 09:58:38 2012] [error] [client 10.101.1.12] [C6kds9y1MVptEDMoOe5KCYha9VWl] [error] [ORM_LONG_QUERY] [/var/etsy/ current/phplib/EtsyORM/Query/RawSql.php:752] [15877310] Query exceeded 10 seconds: long_query_time=83.0927 long_query_string='/* [etsy_shard_005_A] [/ remove_favorite_listing.php] */ DELETE FROM `users_favoritelistings` WHERE `user_id` = ? AND `listing_id` = ?' long_query_trace='#10 __construct() /EtsyModel/ UserFavoriteListingMirror.php:310 #4 delete() /EtsyModel/UserFavoriteListing.php:39 #3 delete() /EtsyModel/User.php:1840 #2 unfavoriteListing() /Controller/ Favorites.php:344 #1 removeFavoriteListingRecord() /Controller/Favorites.php:94 #0 performRemoveFavoriteListing() /var/etsy/current/htdocs/remove_favorite_listing.php: 9', referer: http://www.etsy.com/people/kellanem/favorites?page=5

MySQL: Introspection SQL Comments are awesome!

MySQL: Deletes are expensive * update objects to state=‘deleted’ * use partitions * truncatenator - on ext3, hard link file, move, delete slowly.

Anatomy of a feature: Shop Stats

• Monday, June 18, 12

Anatomy of a feature: Shop Stats

• “Never get into a land war in Asia, and never

build an analytics tool on top of MySQL.

Anatomy of a feature: Shop Stats * buffer writes in Memcache using predictable keys * flush to MySQL tables periodically via cron * bake old data into all possible date ranges, and archived to S3 * truncate tables

bcn.etsy.com: beaconed event stream * Server-side and javascript event stream * At least one per page view * Apache serving static assets * Aggregated on HDFS via logrotate * Archived on S3 * Analyzed via JRuby/Cascading on Hadoop * Doesn’t use: Flume, Scribe, etc

bcn.etsy.com: beaconed event stream

{"event_guid":"c2ffb51808b.6d2be52959ef{".user_id": 8528531,"php_event_name":"s2","php_unique_id":"4fdf1cb5d5c078.37523961","php_event_dat e":"18\/Jun\/2012:08:19:01","locale_currency_code":"USD","pref_language":"en- US","region":"US","detected_region":"US","accept-languages":"en- US,en","isMobileDevice":"0","isMobileSupported":"0","isTabletSupported":"0","isTouch":"0","isEt syApp":"0","listing_ids":[60274277,101504389,98682771,88585080],"cids": [14103953,14239293,14247717,14209614],"query":"blue","keywords": ["blue","blue","blue","blue"],"position":1,"replay_number":1,"s2_cached": 1,"php_ab_test_names":"orm_record_instance_caching;mobile_detector.all_blackberry;multila ng_shops_listings.view;ga_replacement_cookie;disable_search_autosuggest;admin_toolbar;tra nslations.live_translations;ab_analytics_test;search_type_experiment;search_ads.max_replays_ less;search_diversity_experiment;search_cached_listing_cards;placefinder.cache_memcached_ migration;search_stream_a;search_all_items_ignores_supplies;search_default_type;search.two _cluster_deploy;search_parameter_sample;thrift_category2_transform;search.similar_listing_b rowse_page;orm_replicant_safe_find_many;bottom_first;foreign_language_carousel;search.rel ated_searches_all_items;weddings.srp_promos;search_log_page_position;newrelic;clientlog;go

• gle_analytics_async;personalized_endpoint;search_no_dropdown;community_nav_popout;se

curity_settings;search_changes_tooltip;inline_listing_hearts;framelogger;log_normal;analytics_ second_beacon;analytics_second_beacon_privileged;analytics_second_beacon_mobile","php_a b_var_names":"1;1;1;1;control;1;0;A;ponycorn_v3;1;threshold_off;1;1;1;0;all_sans_supplies; 0;1;1;1;1;0;top;0;0;1;0;1;0;1;1;1;0;1;1;1;0;1;0;1","php_ab_selector_names":"

Search Master Search Slave01 Search Slave02 Search SlaveNN BitTorrent to distribute indexes Web01 Web02 WebNN 100% of all indexes

• n each slave

Thrift, with server affinity to improve cache hit ratio, just returns ids databases and memcache hydrate IDs via multi-get, ignore a few failures denormalized listing store, transition from MySQL to Hbase, not user facing pull via cron, push via gearman incremental index, every 7 minutes, avoid even numbered cron times

Search

Search

* Solr trunk * Custom ranking via crunched datasets * BitSet fields for personalized search * Scaling the JVM * 32% of visits, 40% of sales * Also powers categories, unshardable queries * Next time, just use HTTP * Up next: custom codecs * Avoiding sharding

Search

* JVM slow start * Search deployinator does rolling restart * HotSpot and GC causes unpredictable throughput * Overfetch - ask multiple servers, go with 1st response * Index size is important. Don’t store too much.

Photos

* 400 million photos * Uploaded locally, then streamed to S3 * GraphicsMagick FTW * Working set is tiny, served

• ut of Squid

* 2% read failure rate during full S3 outage. * 0% write failure rate during full S3 outage.

Technology no longer part of the stack

* Python Twisted * PostgreSQL and stored procedures * Scala and MongoDB * Clojure and Tokyo Tyrant * Rails * ActiveMQ * RabbitMQ * a "Routes" framework * building RPMs * Lighttpd

Take aways

• 1. A few simple, boring, well known

components

• 2. Extensive instrumentation
• 3. Rapid iteration and feedback loops
• 4. Human centric
• 5. A few tweaks on the classics for scale
• 6. Technology supports business goals

Questions?

More info: http://codeascraft.etsy.com http://slideshare.net/etsy http://github.com/etsy http://www.etsy.com/jobs kellan@etsy.com