Blocks Designing for Scale and Innovation for the worlds largest - - PowerPoint PPT Presentation

Confidential Slide 1

Building Blocks and Stumbling Blocks

Designing for Scale and Innovation for the world’s largest ticket marketplace

Charlie Fineman

November 16, 2011 2

• 5M orders on 1.5-2M Active listings
• About 45K events
• 20% YoY growth in sales
• A typical hour sees us serving about 6-800k complex

pages and about 2M api calls (more on this later) powering affiliate sites around the web.

• We’re also a lot smaller shop!

We’re not the size of mother eBay but…

November 16, 2011 3

Of course site performance but also…

• Operational efficiency
• Developer efficiency

Scaling everything

November 16, 2011 4

• Buyers demand up-to-date inventory… buyer frenzy
• Sellers are constantly updating prices and quantity…

they are playing the market on and off StubHub

• Tickets are not commodities

So as you might imagine the tickets table gets hammered.

An active marketplace

November 16, 2011 5

To Serve and Protect…

Web Bulk POS Manager Tickets β Find My Account Public Feed α Internal External

November 16, 2011 6

Slave

Change Data Capture

10G

Master Lucene/ SOLR Catalog Service

SEDA

Protocol Buffers

Memcache Broker Broker Listener

Double click into LCS

November 16, 2011 7

The Tailored Shopping Experience

November 16, 2011 8

SEDA in the Master: Graceful degradation

Consume Subscribe Load Cascade Serialize Store Notify

AMQ memcached eCom

java.util.ThreadPoolExecutor A useful retrospective on applicability

http://matt-welsh.blogspot.com/2010/07/retrospective-on-seda.html

November 16, 2011 9

• Fast
• Flexible POTENT query language
• Full-text search
• Geo-Spatial added this year
• Various output formats

The Good

• Not great for high frequency writes
• Remember that “active marketplace” comment?
• Flat Data structures (well… there are lists)
• Replication feels like an afterthought (so we built
• ur own)
• Search scores are relative, not absolute so not as

good for “scrubbing”

The Bad

Good and Bad of SOLR

November 16, 2011 10

Double Click into Browse (DCL)

Session Management

• Async preperation (e.g. offers)

Determine Content

• Choose content assembly
• Localization

Generate Context Document

• Url mapping defines the contextual IDs

Render Content

• Translate context document
• Can augment with service calls

November 16, 2011 11

• Efficiently handling the nominal cases

– Customer self service – Getting payments out quicker – Electronic integration

• Primary
• Point-of-(re)sale
• And the out-of-ordinary

– Fraud – BPM

Scaling Operationally

November 16, 2011 12

High level domain model

Fulfillment User Content Catalog

Contact ($) Profile

sell.sh.com buy.sh.com myaccount.sh.com www.sh.com publicfeed.sh.com

November 16, 2011 13

approved purchased confirmed unconfirmed

Order Lifecycle made life easy(er)

fulfilled

Email seller Email buyer Capture $ Create Seller Pmnt

Nominal FedEx

TeamWorks TeamWorks TeamWorks

November 16, 2011 14

approved purchased confirmed unconfirmed

Order Lifecycle made life easy(er)

fulfilled

Barcode Reissue

Electronic Fulfillment

Auto Confirm Disburse PDF

November 16, 2011 15

Fraud Eval

approved purchased confirmed unconfirmed

Order Lifecycle made life easy(er)

fulfilled

Barcode Reissue

Fraud Avoidance

Auto Confirm Disburse PDF

November 16, 2011 16

• Integration with “external book of record for inventory”.
• Uses “two phase commit”

1)

Reserve ticket on external system

2)

Create entry in StubHub book of record

3)

Commit purchase on external system.

• Proof of existence with Ticket Technology.

– Looking to generalize (in both directions) in the future.

Real-time Order Integration

November 16, 2011 17

Splunk and Dye – A perfect match

Web App Log4j (MF) MDC Dye Filter Http Client Dye in Http Header (Dyed?) Http Req Splunk This REALLY improved our triage/debugging procedures

November 16, 2011 18

• Tend to prefer thin frameworks… makes devs think

about what they are doing

– Hibernate

• Interesting observation: new schema vs existing

– Component-based web frameworks

• It’s great to have a legacy but it can really suck
• Don’t short-change CM and deployment.
• Invest in continuous improvement.

Other missives

Confidential Slide 19

Thanks!

cfineman@stubhub.com