GraphWalker: An I/O-Efficient and Resource-Friendly Graph Analytic - - PowerPoint PPT Presentation

▶
graphwalker an i o efficient and resource friendly graph
SMART_READER_LITE
LIVE PREVIEW

GraphWalker: An I/O-Efficient and Resource-Friendly Graph Analytic - - PowerPoint PPT Presentation

Aug 20, 2022 317 likes •571 views

GraphWalker: An I/O-Efficient and Resource-Friendly Graph Analytic System for Fast and Scalable Random Walks Rui Wang , Yongkun Li , Hong Xie , Yinlong Xu , John C.S. Lui* University of Science and Technology of China

slide-1
SLIDE 1

GraphWalker: An I/O-Efficient and Resource-Friendly Graph Analytic System for Fast and Scalable Random Walks

Rui Wang†, Yongkun Li†, Hong Xie‡, Yinlong Xu†, John C.S. Lui*

† University of Science and Technology of China ‡ Chongqing University * The Chinese University of Hong Kong

USENIX ATC 2020

slide-2
SLIDE 2

Graph analytics is one of the top 10 data and analytics technology trends [1]

2

[1] Gartner's top 10 data and analytics technology trends for 2019

Recommendation systems Social networks Webpage links Ø

slide-3
SLIDE 3

Ø

3

[2] The 2012 common crawl graph. http://webdatacommons.org.

Ø

  • Random walks can realize an approximate calculation on large graphs.
slide-4
SLIDE 4

Ø

a) b) c)

4

Massive walks situations! Ø

slide-5
SLIDE 5

Ø

5

Disk Memory Loop until all walks finished ① load a sub-graph to memory ② update walks in the sub-graph

slide-6
SLIDE 6

Ø

6

How many walkers in each loaded sub-graph? How many steps each walker can update?

slide-7
SLIDE 7

Ø

  • 7

walks distribution among subgraphs after 4 steps I/O 𝑣𝑢𝑗𝑚𝑗𝑨𝑏𝑢𝑗𝑝𝑜 =

#,-./ ./0.- #12134 ./0.- 56 3 -,70839:

Some loaded blocks contain only few walkers, results in low I/O utilization

Start 106 walks from a source in Friendster (68.3M vertices) on DurnkardMob

More than 200000 walks Only 12 walks 0.03%

slide-8
SLIDE 8

Ø

  • 8

3 4 5 1 2 7 6 8 9

block b0 block b2 block b1

𝑥<𝑥= 𝑥>𝑥?

3 4 5 1 2 7 6 8 9

𝑥< 𝑥= 𝑥> 𝑥?

block b0 block b2 block b1 Let the most walkers get moved by an I/O

slide-9
SLIDE 9

Ø

  • 9

Many walks remain in the current subgraph after walking one step Synchronized walk updating leads to low walk updating rate

walk steps update rate in each I/O walks in 1st subgraph after each iteration Start 106 walks of 10 steps from a source in Friendster (68.3M vertices) on DurnkardMob

Avg: 0.2%

slide-10
SLIDE 10

10

Ø

Successively update each walk until it moved out of current subgraph Maximize the I/O utilization

  • f a loaded subgraph

Straggler problem: probabilistic approach

with a probability p, we choose to load the subgraph with the shortest walker

slide-11
SLIDE 11

Ø

  • 11

#walk limited as its hard to flush all walks to disk for too many files

walk arrays

1 … 128

bucket array

P−1 256 |v| |v|-128 …

Too many walk arrays Ø

slide-12
SLIDE 12

Ø

  • 12

Dynamic arrays bring high cost for storing walk data

Frequent memory re-allocation p

  • Dynamic walk arrays

1 … 128

bucket array

P−1 256 |v| |v|-128 …

memory waste

slide-13
SLIDE 13

13

Fixed-length block array Walk pool in disk file 1 … P Block array … block 0 block 1 block P source current step 13 39 63 14 40 low s𝐮𝐩𝐬𝐛𝐡𝐟 𝐝𝐩𝐭𝐮 less memory re-allocation low I/O cost

Ø

slide-14
SLIDE 14

Ø Other optimizations

14

Graph block size configuration Walk-conscious cache strategy Data conflict

  • ptimization in

multi-threads

Ø Prototype system——GraphWalker

Light-weighted blocking More details are in the paper

slide-15
SLIDE 15

Ø

  • Ø

15

Largest dataset

slide-16
SLIDE 16

16

DrunkardMob

Optimize the walk management Ø

Graphene

Fine-grained I/O management Ø

GraFSoft KnightKing

Optimize the walk forwarding process Ø

slide-17
SLIDE 17

17

Ø Performance of random walks with different number of walks

  • Fix walk length as 10

GraphWalker achieves 16x-70x speedup. GraphWalker is also capable to support huge graphs and massive walks.

GraphWalker finishes running 1010 walks on the largest dataset CrawlWeb within around one hour.

slide-18
SLIDE 18

18

Ø Performance of random walks with different walk lengths

  • Fix the number of walks as 105

GraphWalker achieves even more than three orders of magnitude in the best case. GraphWalker also achieves 7 – 10x speedup for Kron30.

slide-19
SLIDE 19

19

Ø I/O utilization and walk updating rate

  • RWD (|V| * 6 walks) in YahooWeb (1.6B vertices)

DrunkardMob needs 150 I/Os and GraphWalker only needs 46 I/Os. GraphWalker achieves 2 – 4x I/O utilization.

slide-20
SLIDE 20

20

Ø Compare with single machine systems Graphene and GraFSoft

  • R * 10

GraphWalker achieves 2 - 40x speedup compared to Graphene. GraphWalker achieves 1 - 37x speedup compared to GraFSoft.

slide-21
SLIDE 21

21

Ø Compare with distributed system KnightKing

  • Run |v| walks, each vertex start one walk
  • Terminate with probability 0.15 in each walk step

GraphWalker (1 node) achieves comparable with KnightKing (8 nodes).

slide-22
SLIDE 22

Ø

  • Ø

Ø https://github.com/ustcadsl/graphwalker

22

slide-23
SLIDE 23

23