Arpeggio: Metadata Searching and Content Sharing with Chord Austin - - PowerPoint PPT Presentation

▶
arpeggio
SMART_READER_LITE
LIVE PREVIEW

Arpeggio: Metadata Searching and Content Sharing with Chord Austin - - PowerPoint PPT Presentation

Aug 18, 2023 257 likes •768 views

Arpeggio: Metadata Searching and Content Sharing with Chord Austin T. Clements, Dan R. K. Ports, David R. Karger { aclements, drkp, karger } @mit.edu MIT Computer Science and Artificial Intelligence Laboratory Outline Overview Searching

slide-1
SLIDE 1

Arpeggio:

Metadata Searching and Content Sharing with Chord

Austin T. Clements, Dan R. K. Ports, David R. Karger

{aclements, drkp, karger}@mit.edu

MIT Computer Science and Artificial Intelligence Laboratory

slide-2
SLIDE 2

Outline

Overview Searching Index Gateways Content Distribution Conclusion

Arpeggio – p.1

slide-3
SLIDE 3

The Content-Sharing Problem

Goals Find files matching a search query Identify sources for a file Want full decentralization Assumptions Only searching metadata Metadata is small (compared to actual data) Highly dynamic and unstable network topology, content, and sources

Arpeggio – p.2

slide-4
SLIDE 4

DHTs (Almost) to the Rescue

Great for lookup-by-name Insufficient for efficient search-by-content Powerful underlying LOOKUP abstraction

Arpeggio – p.3

slide-5
SLIDE 5

Outline

Overview Searching Index Gateways Content Distribution Conclusion

Arpeggio – p.4

slide-6
SLIDE 6

Index Entries

(debian, disk1, iso) name Debian Disk1.iso file ID cdb79ca3db1f39b1940ed5... size 586MB type application/x-iso9660-image . . .

Arpeggio – p.5

slide-7
SLIDE 7

Distributed Inverted Indexing

(debian, disk1, iso) name Debian Disk1.iso fi le ID cdb79ca3db1f3...

▽Arpeggio – p.6

slide-8
SLIDE 8

Distributed Inverted Indexing

(debian, disk1, iso) name Debian Disk1.iso fi le ID cdb79ca3db1f3...

debian disk1 iso

▽Arpeggio – p.6

slide-9
SLIDE 9

Distributed Inverted Indexing

(debian, disk1, iso) name Debian Disk1.iso fi le ID cdb79ca3db1f3... (debian, disk2, iso) name Debian Disk2.iso fi le ID 5ccbf54d7e502...

debian disk1 iso disk2

▽Arpeggio – p.6

slide-10
SLIDE 10

Distributed Inverted Indexing

(debian, disk1, iso) name Debian Disk1.iso fi le ID cdb79ca3db1f3... (debian, disk2, iso) name Debian Disk2.iso fi le ID 5ccbf54d7e502... (disk1, freebsd, iso) name Freebsd Disk1.iso fi le ID fbcbfdff31f27de...

debian disk1 iso disk2 freebsd

▽Arpeggio – p.6

slide-11
SLIDE 11

Distributed Inverted Indexing

(debian, disk1, iso) name Debian Disk1.iso fi le ID cdb79ca3db1f3... (debian, disk2, iso) name Debian Disk2.iso fi le ID 5ccbf54d7e502... (disk1, freebsd, iso) name Freebsd Disk1.iso fi le ID fbcbfdff31f27de...

debian disk1 iso disk2 freebsd “debian disk1?”

▽Arpeggio – p.6

slide-12
SLIDE 12

Distributed Inverted Indexing

(debian, disk1, iso) name Debian Disk1.iso fi le ID cdb79ca3db1f3... (debian, disk2, iso) name Debian Disk2.iso fi le ID 5ccbf54d7e502... (disk1, freebsd, iso) name Freebsd Disk1.iso fi le ID fbcbfdff31f27de...

debian disk1 iso disk2 freebsd “debian disk1?” { , } ∩ { , }

▽Arpeggio – p.6

slide-13
SLIDE 13

Distributed Inverted Indexing

(debian, disk1, iso) name Debian Disk1.iso fi le ID cdb79ca3db1f3... (debian, disk2, iso) name Debian Disk2.iso fi le ID 5ccbf54d7e502... (disk1, freebsd, iso) name Freebsd Disk1.iso fi le ID fbcbfdff31f27de...

debian disk1 iso disk2 freebsd “debian disk1?” { , } ∩ { , } Problem: Network hosage

Arpeggio – p.6

slide-14
SLIDE 14

Index-Side Filtering

Keywords are small, so store keywords in index Pick one index node Send full query Index node performs filtering and returns only relevant results Can also include other filterable metadata, e.g. file size, MP3 bitrate, etc.

▽Arpeggio – p.7

slide-15
SLIDE 15

Index-Side Filtering

(debian, disk1, iso) name Debian Disk1.iso fi le ID cdb79ca3db1f3... (debian, disk2, iso) name Debian Disk2.iso fi le ID 5ccbf54d7e502... (disk1, freebsd, iso) name Freebsd Disk1.iso fi le ID fbcbfdff31f27de...

debian disk1 iso disk2 freebsd

▽Arpeggio – p.7

slide-16
SLIDE 16

Index-Side Filtering

(debian, disk1, iso) name Debian Disk1.iso fi le ID cdb79ca3db1f3... (debian, disk2, iso) name Debian Disk2.iso fi le ID 5ccbf54d7e502... (disk1, freebsd, iso) name Freebsd Disk1.iso fi le ID fbcbfdff31f27de...

debian disk1 iso disk2 freebsd “debian disk1?”

▽Arpeggio – p.7

slide-17
SLIDE 17

Index-Side Filtering

(debian, disk1, iso) name Debian Disk1.iso fi le ID cdb79ca3db1f3... (debian, disk2, iso) name Debian Disk2.iso fi le ID 5ccbf54d7e502... (disk1, freebsd, iso) name Freebsd Disk1.iso fi le ID fbcbfdff31f27de...

debian disk1 iso disk2 freebsd “debian disk1?” { }

▽Arpeggio – p.7

slide-18
SLIDE 18

Index-Side Filtering

(debian, disk1, iso) name Debian Disk1.iso fi le ID cdb79ca3db1f3... (debian, disk2, iso) name Debian Disk2.iso fi le ID 5ccbf54d7e502... (disk1, freebsd, iso) name Freebsd Disk1.iso fi le ID fbcbfdff31f27de...

debian disk1 iso disk2 freebsd “debian disk1?” { } Problem: Poor query load-balancing

Arpeggio – p.7

slide-19
SLIDE 19

Keyword-Set Indexing

Build index on keyword sets rather than keywords Store subsets of size ≤ K More keyword-set indexes, but each is shorter Single-keyword indexes are less important, so can be truncated < 29% of web searches have only 1 keyword. [Reynolds & Vahdat 2003] To search: send filtered query to any K-size subset index

▽Arpeggio – p.8

slide-20
SLIDE 20

Keyword-Set Indexing

(debian, disk1, iso) name Debian Disk1.iso fi le ID cdb79ca3db1f3...

(K = 2) debian disk1 iso debian disk1 debian iso disk1 iso

▽Arpeggio – p.8

slide-21
SLIDE 21

Keyword-Set Indexing

(debian, disk1, iso) name Debian Disk1.iso fi le ID cdb79ca3db1f3... (debian, disk2, iso) name Debian Disk2.iso fi le ID 5ccbf54d7e502...

(K = 2) debian disk1 iso debian disk1 debian iso disk1 iso disk2 debian disk2 disk2 iso

▽Arpeggio – p.8

slide-22
SLIDE 22

Keyword-Set Indexing

(debian, disk1, iso) name Debian Disk1.iso fi le ID cdb79ca3db1f3... (debian, disk2, iso) name Debian Disk2.iso fi le ID 5ccbf54d7e502... (disk1, freebsd, iso) name Freebsd Disk1.iso fi le ID fbcbfdff31f27de...

(K = 2) debian disk1 iso debian disk1 debian iso disk1 iso disk2 debian disk2 disk2 iso . . .

▽Arpeggio – p.8

slide-23
SLIDE 23

Keyword-Set Indexing

(debian, disk1, iso) name Debian Disk1.iso fi le ID cdb79ca3db1f3... (debian, disk2, iso) name Debian Disk2.iso fi le ID 5ccbf54d7e502... (disk1, freebsd, iso) name Freebsd Disk1.iso fi le ID fbcbfdff31f27de...

(K = 2) debian disk1 iso debian disk1 debian iso disk1 iso disk2 debian disk2 disk2 iso . . . “debian disk1?”

▽Arpeggio – p.8

slide-24
SLIDE 24

Keyword-Set Indexing

(debian, disk1, iso) name Debian Disk1.iso fi le ID cdb79ca3db1f3... (debian, disk2, iso) name Debian Disk2.iso fi le ID 5ccbf54d7e502... (disk1, freebsd, iso) name Freebsd Disk1.iso fi le ID fbcbfdff31f27de...

(K = 2) debian disk1 iso debian disk1 debian iso disk1 iso disk2 debian disk2 disk2 iso . . . “debian disk1?”

Arpeggio – p.8

slide-25
SLIDE 25

Indexing Cost

m = metadata keywords K = maximum subset size parameter I(m) = index entries =

K

  • i=1

m i

  • =
  • 2m − 1

if m ≤ K

O(mK)

if m > K

▽Arpeggio – p.9

slide-26
SLIDE 26

Indexing Cost

I(m) =

K

  • i=1

m i

  • =
  • 2m − 1

if m ≤ K

O(mK)

if m > K

1 2 4 8 16 32 64 128 256 512 1 2 3 4 5 6 7 8 9 I(m) m K = ∞ K = 4 K = 3 K = 2 K = 1

▽Arpeggio – p.9

slide-27
SLIDE 27

Indexing Cost

I(m) =

K

  • i=1

m i

  • =
  • 2m − 1

if m ≤ K

O(mK)

if m > K For files with many metadata keywords, I(m) is polynomial in m.

Arpeggio – p.9

slide-28
SLIDE 28

Storage Costs (FreeDB)

Number of songs 21,195,244 Total index entries (K = 1) 134,403,379 Index entries per song (K = 1) 6.274406 Total index entries (K = 3) 1,494,688,373 Index entries per song (K = 3) 66.078093

⇒ Total storage cost only an order of magnitude more

than required for K = 1 inverted index.

Arpeggio – p.10

slide-29
SLIDE 29

Choosing K

Larger K improves query load distribution, increases indexing costs

1 10 100 1000 10000 100000 2 4 6 8 10 12 14

Relative index size

K

For web searches: average query length 2.53

Arpeggio – p.11

slide-30
SLIDE 30

Outline

Overview Searching Index Gateways Content Distribution Conclusion

Arpeggio – p.12

slide-31
SLIDE 31

Index Gateways

S1 MF I1: a MF I2: b MF I3: a b

Each file has one metadata block, stored in I(m) indexes.

▽Arpeggio – p.13

slide-32
SLIDE 32

Index Gateways

S1 S2 MF I1: a MF I2: b MF I3: a b

Each peer sharing the file will insert the same metadata block into each index.

▽Arpeggio – p.13

slide-33
SLIDE 33

Index Gateways

S1 S2 MF I1: a MF I2: b MF I3: a b S3

Total insertion cost for m metadata keywords and s source peers: sI(m) messages. Problem: expensive and redundant

▽Arpeggio – p.13

slide-34
SLIDE 34

Index Gateways

Solution: aggregate updates at an index gateway Receives metadata blocks from sources and sends to indexes only when necessary

G S1 MF I1: a MF I2: b MF I3: a b

Insertion cost is now s + I(m) (vs. sI(m))!

▽Arpeggio – p.13

slide-35
SLIDE 35

Index Gateways

Solution: aggregate updates at an index gateway Receives metadata blocks from sources and sends to indexes only when necessary

G S1 S2 MF I1: a MF I2: b MF I3: a b

Insertion cost is now s + I(m) (vs. sI(m))!

▽Arpeggio – p.13

slide-36
SLIDE 36

Index Gateways

Solution: aggregate updates at an index gateway Receives metadata blocks from sources and sends to indexes only when necessary

G S1 S2 MF I1: a MF I2: b MF I3: a b S3

Insertion cost is now s + I(m) (vs. sI(m))!

Arpeggio – p.13

slide-37
SLIDE 37

Outline

Overview Searching Index Gateways Content Distribution Conclusion

Arpeggio – p.14

slide-38
SLIDE 38

Direct Storage?

Content is large Network has churn Kazaa median session length 2.4 minutes [Gummadi et al. 2003] Problem: DHT storage of content is impractical

Arpeggio – p.15

slide-39
SLIDE 39

Indirect Storage

Add indirection Store only small pointers in the network keywords

  keyword-set index search

file IDs

 

  • sources

Arpeggio – p.16

slide-40
SLIDE 40

Content-Sharing Subrings

How to identify sources for a file? Simple solution: “tracker node” Instead, file sources form subrings To find source, LOOKUP random ID in file’s subring Search and maintenance costs same as with tracker, but distributed over network No single point of failure

Arpeggio – p.17

slide-41
SLIDE 41

Outline

Overview Searching Index Gateways Content Distribution Conclusion

Arpeggio – p.18

slide-42
SLIDE 42

Conclusion

Supports search with distributed keyword-set index Extends standard DHT interface with network-side processing Index-side filtering Index gateways Content distribution with indirect storage Indexing via subrings

Arpeggio – p.19

slide-43
SLIDE 43

Arpeggio – p.20

slide-44
SLIDE 44

Bonus Section

Arpeggio – p.21

slide-45
SLIDE 45

Bonus 1 – Postfetching

Indirect storage leads to unavailable content Long metadata expiration leads to visible, unavailable content Insert request blocks into the network When source node rejoins and locates request blocks actively push requested content into the caches of randomly-selected nodes.

Arpeggio – p.22

slide-46
SLIDE 46

Bonus 2 – Metadata Expiration

File availability constantly changes as peers join/leave Expiration rather than polling Peers must be able to handle references to unavailable files Long expiration times to track access attempts for unavailable files

Arpeggio – p.23

slide-47
SLIDE 47

Bonus 3 – Index Replication

Replication instead of erasure coding Only weak consistency required Updates propagated periodically Expirations performed independently

Arpeggio – p.24

slide-48
SLIDE 48

Bonus 4 – Segmentation

At this level, content is atomic Can’t share partially downloaded content Problem: Doesn’t utilize upload bandwidth

▽Arpeggio – p.25

slide-49
SLIDE 49

Bonus 4 – Segmentation

Split content into chunks and share on the chunk level Typical file sharing networks contain many similar files Problem: Underutilization of content sources

▽Arpeggio – p.25

slide-50
SLIDE 50

Bonus 4 – Segmentation

Place chunk boundaries based on content data Identify chunks by hash of their contents Files are now just a list of their constituent chunk IDs

keywords ? ? ykeyword-set index search fi le IDs ? ? ystandard DHT lookup chunk IDs ? ? ycontent-sharing subring sources

Arpeggio – p.25