Exploiting player behavior in distributed architectures for online - - PowerPoint PPT Presentation

Problem statement P2P routing Data partitioning Conclusion

Exploiting player behavior in distributed architectures for online games

Ph.D. defense of Sergey Legtchenko

INRIA/LIP6/UPMC/CNRS

Supervision: S´ ebastien Monnet Pierre Sens

1 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Massively Multiplayer Online Games (MMOGs)

1998 2012 2000 2002 2004 2006 2008 2010

2 4 6 8 10 12 14 16 18 20 23

Subscriptions in millions Date Cumulated MMOG population

Market: $2.7 billions in 2010

2 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

MMOGs rely on expensive large-scale infrastructures

Datacenter-based:

1000’s of server-blades 100’s of terabytes of DRAM Up to 80% of the financial revenue

[Kesselman’05]

3 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Problem: architectures are static, workload is dynamic

Normal load Server 2 Server 3 Server 1 Normal load Normal load

Static game partitioning unadapted to player density evolutions

4 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Problem: architectures are static, workload is dynamic

Normal load Server 2 Server 3 Server 1 Normal load Normal load

Event

Static game partitioning unadapted to player density evolutions

4 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Problem: architectures are static, workload is dynamic

Normal load Server 2 Server 3 Server 1 Normal load Normal load

Event

Static game partitioning unadapted to player density evolutions

4 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Problem: architectures are static, workload is dynamic

Normal load Server 2 Server 3 Server 1 Overloaded Underloaded

Event

Static game partitioning unadapted to player density evolutions

4 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Consequence: high cost, low efficiency

Current MMOGs:

Lots of empty servers [Cheslack-Postava et al, USENIX’12] Lots of overloaded servers [Varvello et al, NetGames’09] Independent game instances limited to few 100’s of players Low quality of service No geo-scale seamless virtual universe No epic battles

5 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Academic research on MMOGs

Extensive efforts on adaptative mechanisms:

Load balancing Interest management

Why no impact? lack of robustness/performance

l a r g e s c a l e M M O G s fast paced MMOGs

State of the art: Peer-to-peer (p2p)

[Colyseus, NSDI'06] [Donnybrook, SIGCOMM'08] [Solipsis, PDPTA'03] [Hydra, NetGames'07]

Server based

[Sirikata, USENIX'12] [Najaran et al., NetGames'10] [ALVIC-NG, NetGames'08]

Hybrid

[Jardine et al., NetGames'08] [Walkad, IPTPS'09]

Well suited for:

6 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Contributions of the Thesis

Guideline: improving MMOGs by making them aware of player behavior Improving robustness:

BlueBanana: increasing resilience of p2p MMOGs to player movement [DSN10]

Improving performance:

DONUT: improving routing in large-scale p2p MMOGs with heterogeneous peer distributions [SRDS11] QuakeVolt: Efficient data management in server-based MMOGs with strong latency requirements [ongoing work]

7 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Outline of the talk

Focusing on performance improvement:

Part 1: Improving routing in p2p MMOGs with heterogeneous peer distributions with DONUT (approx 20 minutes) Part 2: efficient data management for large-scale virtual battlegrounds with QuakeVolt (approx 15 minutes)

8 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Part 1: Improving routing in peer-to-peer MMOGs

8 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Context: large-scale p2p MMOGs

Nearest-neighbor overlays Useful properties:

Data locality Greedy routing Cheap Good fault resilience

[Mercury, VON, VoroNet, RayNet] Nearest-neighbors p2p overlays:

Peer A

2D game partition with associated overlay graph

Z

• n

e

• f

A 9 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Problem: lack of routing efficiency

Efficient greedy routing:

O(logd(N)) with Small-World shortcuts

Kleinberg, STOC ′2000

Requires estimation of hop distances between peers

Peer A F C D E Game space Peer B

?

No global topology knowledge Peer A has:

Knowledge of direct neighbors

10 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Estimating hop distance

Peer B Peer R

d (R,B)

hops

Uniform distribution: easy

d (R,B)

euclid

proportional to

d

hops euclid

d

Peer B Peer R

O G

Heterogeneous distribution: hard depends on density

d

hops

d (R,B) through G: 5 hops d (R,B) through O: 2 hops d (R,B): 4 hops

hops hops hops

11 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Efficiency required despite heterogeneity

Real distributions: non-uniform

10 20 30 40 50 60 70 5 10 15 20 25 30

Density distribution of Second Life islands: "Isle of View" "Dance"

% of cells Density (Players per cell)

Routing in MMOGs:

Joins Player teleportation

P joins position c bootstrap: peer B

c

B

12 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Efficiency required despite heterogeneity

Real distributions: non-uniform

10 20 30 40 50 60 70 5 10 15 20 25 30

Density distribution of Second Life islands: "Isle of View" "Dance"

% of cells Density (Players per cell)

Routing in MMOGs:

Joins Player teleportation

P joins position c bootstrap: peer B

c

B

12 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Efficiency required despite heterogeneity

Real distributions: non-uniform

10 20 30 40 50 60 70 5 10 15 20 25 30

Density distribution of Second Life islands: "Isle of View" "Dance"

% of cells Density (Players per cell)

Routing in MMOGs:

Joins Player teleportation

P joins position c bootstrap: peer B

c

B

12 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Efficiency required despite heterogeneity

Real distributions: non-uniform

10 20 30 40 50 60 70 5 10 15 20 25 30

Density distribution of Second Life islands: "Isle of View" "Dance"

% of cells Density (Players per cell)

Routing in MMOGs:

Joins Player teleportation

P joins position c bootstrap: peer B

c

B

12 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Efficiency required despite heterogeneity

Real distributions: non-uniform

10 20 30 40 50 60 70 5 10 15 20 25 30

Density distribution of Second Life islands: "Isle of View" "Dance"

% of cells Density (Players per cell)

Routing in MMOGs:

Joins Player teleportation

P joins position c bootstrap: peer B

c

B P

12 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Contribution: DONUT, density-aware shortcut rewiring mechanism

Idea: Make peers “density-aware” On each peer:

Step 1: Dynamically create map of game space density Step 2: Use map to build density-aware small-world shortcuts

13 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Step 1: Making peers “density aware”

Naive solution: Locally store coordinates of all peers

Heterogeneous host distribution

(Second Life trace snapshot)

Problem: not scalable!

Thousands of peers High churn rates Peer mobility

Need to approximate the distribution.

14 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Approximate the density distribution

High density zone 1 High density zone 2

High density Low density Approximation Distribution

15 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Density map creation: two tasks

Task 1: Compute local density Task 2: Exchange density info Locally, using direct neighbors Piggybacking and gossiping

16 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Step 2: Log-partitioning for Small-World

Shortcut link to one random peer in each partition

Obtained distribution enables small-world property...

Girdzijauskas, ICDEW ′05

...if hop distances are accurately approximated

Log partitions of overlay graph by A and Small-World shortcuts of A

A

P , P ,..., P partitions

1 2 log(N)

B P, 2

i

• i

d(A,B)

• i+1

2 with d(A,B) in hops

17 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Hop distance estimation using map

Estimate hop distance between A & B:

1

Find the regions that intersect [AB]

2

Estimate hops in each region

3

Sum hops for all regions

A B

Uniform density inside region

18 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Small-world shortcuts with Monte-Carlo sampling

On local map:

1

Find max hop distance to peer

2

Approximate log partition

Cost:

Depends on sampling precision No remote operations

Position

• f A in

game space Farthest coordinate Hotspot

Step 1: A Finds maximal hop distance

19 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Small-world shortcuts with Monte-Carlo sampling

On local map:

1

Find max hop distance to peer

2

Approximate log partition

Cost:

Depends on sampling precision No remote operations

Max hop distance D

Step 2: A approximates log partitions

D / 2 D / 4

D / 8 D / 16 Each shortcut is inside a distinct log partition Position

• f A in

game space

19 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Simulation based on real data

Latency traces:

collected between 2500 hosts spread over the internet.

Churn traces:

Overnet, Skype, Microsoft corporate desktops

Game space density traces:

Traces derived from Second Life avatar distribution.

Data Credit: Second Life: La et al.] Churn: http://fta.inria.fr/, [Latency: Madhyastha et al., 20 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Comparison: DONUT vs

Non-small-world shortcuts:

Random Uniform shortcut distribution

Small-world shortcuts:

Uniform: no density information Oscar: density sampling with random walks Optimal: each peer holds the overlay graph

21 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

DONUT is close to optimal

5 10 15 20 3 4 5 6 7 8 9 10 Random Oscar Log2 Uniform DONUT Optimal

Overlay size (x1000) Average route length (hops)

within 10% of optimal

22 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

DONUT latency is within 2% of

• ptimal

Overlay size: 2500 peers A v e r a g e l

• k

u p l a t e n c y ( m s ) 100 200 300 400 500 Optimal DONUT Uniform Oscar Random

23 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Collecting global state is cheap

Time (days) 1 2 3 4 5 6 7

E x c h a n g e d i n f

• r

m a t i

• n

( b p s / n

• d

e )

2 4 6 8 12 14 10 Lookup Gossip

Overlay size: 2500 peers

24 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Contributions in p2p MMOGs:

Technique to aggregate distributed game space information.

Cheap: a few bytes per second per peer.

Technique to build density aware shortcuts in the overlay

Improves state-of-the-art by 20% Accurate: within 10% of optimal

25 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Part 2: MMOGs with tight latency requirements

Work in progress!

25 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Context: server-based MMOFPS

Large scale First Person Shooters

Partition-based design:

Game progression in zone A Game progression in zone B 2 tasks per server: Game progression View propagation 26 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Context: server-based MMOFPS

Large scale First Person Shooters

Partition-based design:

Game progression in zone A Game progression in zone B 2 tasks per server: Game progression View propagation state transfer 26 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Context: server-based MMOFPS

Large scale First Person Shooters

Partition-based design:

Game progression in zone A Game progression in zone B 2 tasks per server: Game progression View propagation 26 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Problem: scalability ruins performance

Scalability mechanism:

Zone Server 1 Zone Server 2 Zone Server 3 Zone Server 4

Logical topology Game space

Zone 1 Zone 3 Zone 2 Zone 4

Density Hotspot Overloaded Overloaded

Ensures scalability...

Fair load balancing Elastic horizontal scalability

...but fragments game space:

Increase of inter-zone transfers Increase of network traffic

27 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Problem: scalability ruins performance

Scalability mechanism:

Zone Server 1

Zone Server 2

Zone Server 3 Zone Server 4

Logical topology Game space

Zone 1 Zone 3

Zone 2

Zone 4

Zone 5

Zone 6 Zone 7 Zone Server 7 Zone Server 6 Zone Server 5

Ensures scalability...

Fair load balancing Elastic horizontal scalability

...but fragments game space:

Increase of inter-zone transfers Increase of network traffic

27 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

What to do?

Fragmentation is harmful, so how to:

Ensure scalability Limit fragmentation

Contributions:

Scalability analysis of Quake III, a popular server-based game Scalable architecture that limits game space fragmentation

28 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Quake III First Person Shooter

Not quite an MMOFPS:

Single server Limited to less than 100 players

Benefits of using Quake III:

Still popular and open source High responsiveness requirements Design similar to many MMOG servers

F r a m e c a l c u l a t i

• n

Q3 Server execution tasks

Receive client events Update object states Compute player views Send player views

20 frames per second (fps) when not overloaded

View propagation Game progession 29 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Quake III bandwidth consumption

5 1 1 5 2 2 5 1 2 3 4 5 6 7 8 B a n d w i d t h ( K b i t / s ) C l i e n t s O u t I n

View propagation

Game progression

30 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Quake III CPU consumption/framerate

• 19. 2
• 19. 3
• 19. 4
• 19. 5
• 19. 6
• 19. 7
• 19. 8
• 19. 9

20

• 20. 1

FPS

• 19. 1

20 40 60 80 100 10 20 30 40 50 60 70 80 C P U l

• a

d ( % ) Clients

(Lower is better) (Higher is better)

31 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

QuakeVolt MMOG architecture

Result of scalability analysis: Game progression scales much better than view propagation Idea: Decouple Game progression from View propagation 3-tier architecture:

Quake III server ensures Game progression Low latency (in-memory) database stores game state View propagation is delegated to a set of Snapshot Mirrors

32 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Benefits of the design

Eases dynamic adaptation to workload:

Limited data fragmentation Easy dynamic reconfiguration Elastic scale out No client-side modification

Game server in-memory DB

SM1 SM2 SM3

Player events Player views

R a n g e q u e r i e s Game state propagation 33 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Distributed frame computation

Game server in-memory DB

Start of frame n End of frame n

Frame n+1

Game progression Wait Player view computation

Player events

P l a y e r v i e w s game state not ready rangeGet frame n rangeGet frame n frame n frame n consumed? yes

(1) (1) (2) (2) (3) (4) (3) (4) (5) (5) (6) (6)

GS DB SM1 SM2 SM3 SM1 SM2 SM3

34 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Promising work

VoltDB used as in-memory database Runs on our cluster and on Grid’5000 with 10’s of clients Native protocol unmodified (legacy Q3 clients can connect) Already playable (and fun!) Easy to implement (modification of < 0.03% of Q3 code)

35 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Conclusion

35 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Integration of player behavior is good for MMOG architectures

Contribution summary

1

Better robustness of p2p MMOGs through prediction of player trajectories (not detailed in the talk)

2

Better routing performance in p2p MMOGs thanks to player distribution monitoring

3

Less data fragmentation in server-based MMOGs thanks to accurate player view management

Behavior-aware mechanisms are lightweight and generic

36 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Future work and perspectives

Short term:

DONUT on top of existing overlays for range querying Extensive evaluation of QuakeVolt Adaptative mechanisms for QuakeVolt elasticity and scale out

Long term: exploiting player behavior for future MMOGs

Improvement of player behavior modeling Design of enhanced monitoring techniques Better integration of player behavior at systems level

37 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games

Problem statement P2P routing Data partitioning Conclusion

Thank you for your attention!

Publications related to subject:

DONUT: Building Shortcuts in Large-Scale Decentralized Systems with Heterogeneous Peer Distributions. S.Legtchenko,

S.Monnet and P.Sens [SRDS11]

BlueBanana: resilience to avatar mobility in distributed

• MMOGs. S.Legtchenko, S.Monnet and G.Thomas [DSN10]

Other publications:

RelaxDHT: a churn-resilient replication strategy for peer-to-peer distributed hash-tables. S.Legtchenko, S.Monnet,

P.Sens and G.Muller [TAAS11]

Churn-Resilient Replication Strategy for Peer-to-Peer Distributed Hash-Tables. S.Legtchenko, S.Monnet, P.Sens and

G.Muller [SSS09]

37 / 37 S.Legtchenko - INRIA/LIP6/UPMC/CNRS • Exploiting player behavior in distributed architectures for online games