Search + Discovery Peter Bourgon
Evolution of search Relevance ranking A bit about SOA
EVOLUTION OF SEARCH
KATAMARI DAMACY, NAMCO
HOW TO FIX?
HOW TO FIX? ElasticSearch
HOW TO FIX? ElasticSearch Greenfield development
HOW TO FIX? ElasticSearch Greenfield development Parallel dark launch
HOW TO FIX? ElasticSearch Greenfield development Parallel dark launch Migrate traffic slowly
HOW TO FIX? ElasticSearch Greenfield development Parallel dark launch Migrate traffic slowly Scale out
WHY ELASTICSEARCH
WHY ELASTICSEARCH Good API
WHY ELASTICSEARCH Good API Clear path for growth
WHY ELASTICSEARCH Good API Clear path for growth Batteries included
WHY ELASTICSEARCH Good API Clear path for growth Batteries included Works like you expect it to
COMPONENTS
RELEVANCE RANKING
PageRank
DiscoRank
DISCORANK DETAILS
DISCORANK DETAILS Calculations done offline
DISCORANK DETAILS Calculations done offline Serialize to compact array
DISCORANK DETAILS Calculations done offline Serialize to compact array Load in ES custom scorer
DISCORANK DETAILS Calculations done offline Serialize to compact array Load in ES custom scorer “Hybrid” 2-phase queries
DISCORANK DETAILS Calculations done offline Serialize to compact array Load in ES custom scorer “Hybrid” 2-phase queries Recalculate/reload every N
DISCORANK++
DISCORANK++ + Time component → “Hotness”
DISCORANK++ + Time component → “Hotness” + User behavior → “Trust”
THE BIGGER PICTURE
INDEXING Single-purpose application(s) Stateless and idempotent Build, iterate, iterate, iterate Full catalog ~= 1 hour
SEARCHING Single-purpose application Stateless and idempotent SC/ES DSL translation Open-source ES library
EXPLORING Single-purpose application “Stateless” and idempotent Special, complex searches Cached on regular interval
SERVICE ORIENTED ARCHITECTURE
SOA AT SOUNDCLOUD
SOA AT SOUNDCLOUD Bazooka platform
SOA AT SOUNDCLOUD Bazooka platform API as firewall
SOA AT SOUNDCLOUD Bazooka platform API as firewall 12 Factor applications
12 FACTOR APPS
12 FACTOR APPS Single codebase (repository)
12 FACTOR APPS Single codebase (repository) Config stored in environment
12 FACTOR APPS Single codebase (repository) Config stored in environment Backing services as resources
12 FACTOR APPS Single codebase (repository) Config stored in environment Backing services as resources Stateless
12 FACTOR APPS Single codebase (repository) Config stored in environment Backing services as resources Stateless Horizontal scaling with processes
WHY SOA?
WHY SOA? Velocity
WHY SOA? Velocity Developer happiness
WHY SOA? Velocity Developer happiness Distributed systems are complex
ElasticSearch core + SOA infrastructure
THANKS! (ALSO WE ARE HIRING) Peter Bourgon @peterbourgon http://peter.bourgon.org
Recommend
More recommend
