#GHC18 DROPBOX FULL- TEXT FULL SEARCH PAGE 2 | GRACE HOPPER - - PowerPoint PPT Presentation

Samantha Steele | sammyst@dropbox.com

N A U T I L U S : L E S S O N S F R O M B U I L D I N G D R O P B O X ’ S L A R G E S C A L E , D I S T R I B U T E D S E A R C H E N G I N E

#GHC18

PAGE 2 | GRACE HOPPER CELEBRATION 2018 PRESENTED BY ANITAB.ORG AND THE ASSOCIATION FOR COMPUTING MACHINERY

#GHC 18

DROPBOX FULL- TEXT FULL SEARCH

PAGE 3 | GRACE HOPPER CELEBRATION 2018 PRESENTED BY ANITAB.ORG AND THE ASSOCIATION FOR COMPUTING MACHINERY

#GHC18

#GHC 18

REPLACING DROPBOX’S LEGACY SEARCH SYSTEM

• Built in 2013, served Dropbox well for

many years

• Only scaled to paid users
• Not fmexible for fast experimentation
• Didn’t support modern search features

(I18N Support, Snippets)

PAGE 4 | GRACE HOPPER CELEBRATION 2018 PRESENTED BY ANITAB.ORG AND THE ASSOCIATION FOR COMPUTING MACHINERY

#GHC18

#GHC 18

ELASTICSEARCH EVALUATION

15 Clusters, each composed of:

• 5 Master hosts
• 50 Data hosts, each running 2

ElasticSearch data nodes

• 150 shards with 2x replication for a

total of 300 shards.

PAGE 5 | GRACE HOPPER CELEBRATION 2018 PRESENTED BY ANITAB.ORG AND THE ASSOCIATION FOR COMPUTING MACHINERY

#GHC18

#GHC 18

ELASTICSEARCH EVALUATION

Single cluster P95 Latency corresponds to p75 end to end latency

PAGE 6 | GRACE HOPPER CELEBRATION 2018 PRESENTED BY ANITAB.ORG AND THE ASSOCIATION FOR COMPUTING MACHINERY

#GHC18

#GHC 18

NAUTILUS GOALS

• Scale to all 500 million

users

• Low-latency, 350ms p95
• High availability
• Support fast

experimentation

• Incremental fjeld updates

for metadata indexing

PAGE 7 | GRACE HOPPER CELEBRATION 2018 PRESENTED BY ANITAB.ORG AND THE ASSOCIATION FOR COMPUTING MACHINERY

#GHC18

#GHC 18

NAUTILUS ARCHITECTURE

PAGE 8 | GRACE HOPPER CELEBRATION 2018 PRESENTED BY ANITAB.ORG AND THE ASSOCIATION FOR COMPUTING MACHINERY

#GHC18

#GHC 18

NAUTILUS ENGINE

Nautilus Partitioning Document Sharing within a Partition

PAGE 9 | GRACE HOPPER CELEBRATION 2018 PRESENTED BY ANITAB.ORG AND THE ASSOCIATION FOR COMPUTING MACHINERY

#GHC18

#GHC 18

INFORMATION RETRIEVAL 101

DocID 1, Metadata: T erm Frequency = 152 Positions = 5, 12, 22… Inverted Search Index Single Posting list Entry

PAGE 10 | GRACE HOPPER CELEBRATION 2018 PRESENTED BY ANITAB.ORG AND THE ASSOCIATION FOR COMPUTING MACHINERY

#GHC18

#GHC 18

INVERTED INDEX BUILT ON ROCKSDB

PAGE 11 | GRACE HOPPER CELEBRATION 2018 PRESENTED BY ANITAB.ORG AND THE ASSOCIATION FOR COMPUTING MACHINERY

#GHC18

#GHC 18

OPTIMIZING ROCKSDB PERFORMANCE

Old Posting List Format New Exploded Posting List

PAGE 12 | GRACE HOPPER CELEBRATION 2018 PRESENTED BY ANITAB.ORG AND THE ASSOCIATION FOR COMPUTING MACHINERY

#GHC18

#GHC 18

OPTIMIZING CGO PERFORMANCE

• C Memory != Go Memory
• Cgoroutines != Goroutines
• C to go calls are very

expensive

PAGE 13 | GRACE HOPPER CELEBRATION 2018 PRESENTED BY ANITAB.ORG AND THE ASSOCIATION FOR COMPUTING MACHINERY

#GHC18

#GHC 18

DEBUGGING AVAILABILITY ISSUES

# of partitions served drops as processes repeatedly OOM

PAGE 14 | GRACE HOPPER CELEBRATION 2018 PRESENTED BY ANITAB.ORG AND THE ASSOCIATION FOR COMPUTING MACHINERY

#GHC18

#GHC 18

OOM CASE STUDY 1

Inverted Search Index Classic Memory Leak profjle

PAGE 15 | GRACE HOPPER CELEBRATION 2018 PRESENTED BY ANITAB.ORG AND THE ASSOCIATION FOR COMPUTING MACHINERY

#GHC18

#GHC 18

MEMORY USAGE ON ANOTHER MACHINE

Infmection Point

PAGE 16 | GRACE HOPPER CELEBRATION 2018 PRESENTED BY ANITAB.ORG AND THE ASSOCIATION FOR COMPUTING MACHINERY

#GHC18

#GHC 18

ROOT CAUSE

PAGE 17 | GRACE HOPPER CELEBRATION 2018 PRESENTED BY ANITAB.ORG AND THE ASSOCIATION FOR COMPUTING MACHINERY

#GHC18

#GHC 18

OOM CASE STUDY 2

Sudden Memory Spikes

PAGE 18 | GRACE HOPPER CELEBRATION 2018 PRESENTED BY ANITAB.ORG AND THE ASSOCIATION FOR COMPUTING MACHINERY

#GHC18

Nautilus launched to 100% in June Steady state 99.9% availability 300ms latency p95 Coming soon: Better I18N Support, Snippets and more

CONCLUSIONS

#GHC18

Feel free to reach out: sammyst@dropbox.com