Richard Gibson Ph.D. Thesis Presentation December 6, 2013 Computer - - PowerPoint PPT Presentation

Regret Minimization in Games and the Development of Champion Multiplayer Computer Poker Agents Richard Gibson

Ph.D. Thesis Presentation December 6, 2013

Computer Poker Research Group

Heads Up Limit Texas Hold'em

Source: ebaumsworld.com

Bet! Fold? Call? Raise?

Heads Up No-limit Texas Hold'em

Source: ebaumsworld.com

All-in! Bet!

3-Player Limit Texas Hold'em

Source: ebaumsworld.com

Fold? Call? Raise? Bet! Call.

Source: toonpool.com

3-Player Limit Texas Hold'em

Source: ebaumsworld.com Source: toonpool.com

2010 - 2013

Hyperborean3p

Hyperborean3p

• No theory

– 3-player – Imperfect recall

• Slow
• Memory expensive

2009

Hyperborean3p

• No theory

– 3-player – Imperfect recall

• Slow
• Memory expensive

2009

• New theory

– Many players – Imperfect recall

• Fast
• Improved performance

with limited memory

2013

Outline of Presentation

• Background

– Counterfactual Regret Minimization (CFR)

• Theoretical Advancements for CFR in:

– Many player games – Imperfect recall games

• CFR Speed-Ups
• Tricks with Memory Limitations
• Conclusion + Future Work

Outline of Presentation

• Background

– Counterfactual Regret Minimization (CFR)

• Theoretical Advancements for CFR in:

– Many player games – Imperfect recall games

• CFR Speed-Ups
• Tricks with Memory Limitations
• Conclusion + Future Work

Background - Kuhn Poker

Background - Kuhn Poker

c

Background - Kuhn Poker

c

?

QJ QK 1/6 1/6 ... ...

Background - Kuhn Poker

1 1 b c b c

Check / Bet ? Information set

?

c QJ QK 1/6 1/6 ... ...

Background - Kuhn Poker

1 1 2 b c 2 b c f c f c

Bet! Fold / Call ?

?

c QJ QK 1/6 1/6 ... ...

Background - Kuhn Poker

1 1 2 b c 2 b c f c +1 f c +1

Fold. Bet! +1

• 1

c QJ QK 1/6 1/6 ... ...

Background - Kuhn Poker

1 1 2 b c 2 b c f c +1

• 2

f c +1 +2

Call. Bet! +2 / -2

• 2 / +2

/

c QJ QK 1/6 1/6 ... ...

Background - Kuhn Poker

1 1 2 2 b c 2 2 b c f c +1

• 2

c b f c +1 +2 c b

Check. Check / Bet ?

?

c QJ QK 1/6 1/6 ... ...

Background - Kuhn Poker

1 1 2 2 b c 2 2 b c f c +1

• 2

c b f c +1 +2 c b

Check. Check.

• 1 / +1

/ +1 / -1

+1

• 1

c QJ QK 1/6 1/6 ... ...

Background - Kuhn Poker

1 1 2 2 b c 2 2 b c f c +1

• 2

c b f c +1 +2 c b

Fold / Call ? Bet!

+1

• 1

1 1 f

• 1

+2 c f

• 1
• 2

c

Information set

?

c QJ QK 1/6 1/6 ... ...

Background

1 1 2 2 b c 2 2 b c f c +1

• 2

c b f c +1 +2 c b +1

• 1

1 1 f

• 1

+2 c f

• 1
• 2

c

In general:

Extensive-Form Game

c QJ QK 1/6 1/6 ... ...

Background

1 1 2 2 .4 .6 2 2 .4 .6 .1 .9 +1

• 2

1 1 +1 +2 .8 .2 +1

• 1

1 1 .7

• 1

+2 .3 .7

• 1
• 2

.3

In general:

Extensive-Form Game

Strategy Profile

c QJ QK 1/6 1/6 ... ...

Background

In general:

Extensive-Form Game

Nash Equilibrium Strategy Profile

Nash equilibrium:

“No one can change their strategy and do any better.”

Background

In general:

Extensive-Form Game

Nash Equilibrium Strategy Profile

Nash equilibrium:

“No one can change their strategy and do any better.”

1/3 1/3 1/3

Every game has a Nash equilibrium.

Background

In general:

Extensive-Form Game

Nash Equilibrium Strategy Profile

Nash equilibrium:

“No one can change their strategy and do any better.”

1/3 1/3 1/3

Every game has a Nash equilibrium.

?

Outline of Presentation

• Background

– Counterfactual Regret Minimization (CFR)

• Theoretical Advancements for CFR in:

– Many player games – Imperfect recall games

• CFR Speed-Ups
• Tricks for CFR with Memory Limitations
• Conclusion + Future Work

CFR

1 1 2 2 b c 2 2 b c f c +1

• 2

c b f c +1 +2 c b +1

• 1

1 1 f

• 1

+2 c f

• 1
• 2

c

• “The alpha-beta

search of imperfect information games.”

c QJ QK 1/6 1/6 ... ...

CFR

1 1 2 2 b c 2 2 b c f c +1

• 2

c b f c +1 +2 c b +1

• 1

1 1 f

• 1

+2 c f

• 1
• 2

c

• “The alpha-beta

search of imperfect information games.”

• Offline algorithm

c QJ QK 1/6 1/6 ... ...

CFR

• “The alpha-beta

search of imperfect information games.”

• Offline algorithm
• Iterative, “self-play”

1 1 2 2 .5 .5 2 2 .5 .5 .5 .5 +1

• 2

.5 .5 .5 .5 +1 +2 .5 .5 +1

• 1

1 1 .5

• 1

+2 .5 .5

• 1
• 2

.5 c QJ QK 1/6 1/6 ... ...

CFR

• “The alpha-beta

search of imperfect information games.”

• Offline algorithm
• Iterative, “self-play”
• For each iteration,

update action probabilities at every information set.

1 1 2 2 .3 .7 2 2 .3 .7 1 +1

• 2

1 1 +1 +2 .8 .2 +1

• 1

1 1 .5

• 1

+2 .5 .5

• 1
• 2

.5 c QJ QK 1/6 1/6 ... ...

CFR

Nash Equilibrium Strategy Profile Strategy 1 + Strategy 2 + ... + Strategy T T Average Strategy Profile

=

T

∞

T = number of iterations

Background

Extensive-Form Game

Nash Equilibrium Strategy Profile

CFR

Background

Kuhn Poker

Nash Equilibrium Strategy Profile

CFR

Background

Texas Hold'em

Nash Equilibrium Strategy Profile

CFR

>1014 information sets > 5 million GB

Background

Large Extensive-Form Game

Nash Equilibrium Strategy Profile

?

Background

Large Extensive-Form Game Abstract Game

Background

Abstract Game

• Merge card deals into buckets.

Extensive-Form Game

Background

Abstract Game

• Merge card deals into buckets.

Extensive-Form Game

Background

Extensive-Form Game Abstract Game

>1014

≈109

Background

Extensive-Form Game Abstract Game

>1014

≈109

Abstract Game Equilibrium Strategy

CFR

Background

Extensive-Form Game Abstract Game

>1014

≈109

Abstract Game Equilibrium Strategy

Approximate Full Game Equilibrium Strategy

≈100 GB

CFR

Outline of Presentation

• Background

– Counterfactual Regret Minimization (CFR)

• Theoretical Advancements for CFR in:

– Many player games – Imperfect recall games

• CFR Speed-Ups
• Tricks with Memory Limitations
• Conclusion + Future Work

Theory – Many Player Games

Extensive-Form Game

Nash Equilibrium Strategy Profile

CFR

L I A R !

Theory – Many Player Games

3-or-more Player Game

?

(Not equilibrium)

CFR

2-player Zero-Sum Game

Nash Equilibrium Strategy Profile

CFR

Theory – Many Player Games

3-player Limit Texas Hold'em

Good strategy? (Not equilibrium)

CFR

Agent Total Bankroll (mbb/g) Hyperborean3p 319 ± 2 dpp 171 ± 2 akuma 151 ± 2 CMURingLimit

• 37 ± 2

dcu3pl

• 63 ± 2

Bluechip

• 548 ± 2

Annual Computer Poker Competition 3-Player Limit Texas Hold'em - 2009

Theory – Many Player Games

c 1 1 2 2 2 2 1

• 1

1 QJ QK c b b c c b f c c b f c f c f c 1/6 1/6 +1

• 1

+2 +1 +2

• 1
• 1
• 2

+1

• 2

... ...

Theory – Many Player Games

1 1 2 2 2 2 1

• 1

1 c b b c c b f c c b f c f c f c +1

• 1

+2 +1 +2

• 1
• 1
• 2

+1

• 2

c QJ QK 1/6 1/6 ... ...

Theory – Many Player Games

1 1 2 2 2 2 1

• 1

1 c b b c c b f c c b f c f c f c +1

• 1

+2 +1 +2

• 1
• 1
• 2

+1

• 2

Dominated Strategies

c QJ QK 1/6 1/6 ... ...

Theory – Many Player Games

2 2 2 2 1

• 1

1 c b b c c b f c b c f c f c +1

• 1

+2 +1

• 1
• 1
• 2
• 2

c QJ QK 1/6 1/6 ... ... 1 1

Theory – Many Player Games

1 1 2 2 2 2 1

• 1

1 c b b c c b f c b c f c f c +1

• 1

+2 +1

• 1
• 1
• 2
• 2

Iteratively Dominated Strategy

c QJ QK 1/6 1/6 ... ...

Theory – Many Player Games

Average Strategy Profile T

∞

No Iteratively Dominated Strategies 3-or-more Player Game

CFR New!

[G., arXiv ePrints 2013]

Theory – Many Player Games

Average Strategy Profile T

∞

No Iteratively Dominated Strategies 3-or-more Player Game

CFR New!

“Current” Strategy Profile T Finite T

No Iteratively Dominated Strategies 3-or-more Player Game

CFR New!

[G., arXiv ePrints 2013]

Theory – Many Player Games

3-Player Limit Texas Hold'em - 2012

New!

Outline of Presentation

• Background

– Counterfactual Regret Minimization (CFR)

• Theoretical Advancements for CFR in:

– Many player games – Imperfect recall games

• CFR Speed-Ups
• Tricks with Memory Limitations
• Conclusion + Future Work

Imperfect Recall

Extensive-Form Game Abstract Game

Abstract Game Equilibrium Strategy

CFR

L I A R ! L I A R !

Imperfect Recall

“Imperfect Recall” Abstract Game

?

(Not equilibrium)

CFR

“Perfect Recall” Abstract Game

Abstract Game Equilibrium Strategy

CFR

Imperfect Recall

Pre-flop

Imperfect Recall

Pre-flop Flop

Imperfect Recall

Imperfect Recall Abstract Game

Imperfect Recall

Perfect Recall Abstract Game

Imperfect Recall

Extensive-Form Game

“Well-formed” Imperfect Recall Game

Abstract Game Equilibrium Strategy

CFR

• Unfortunately, does not prove

convergence for our best poker abstractions.

• However, this was used in diabetes

treatment research

[Chen and Bowling, NIPS 2012]. [Lanctot, G., Burch and Bowling, ICML 2012]

New!

Outline of Presentation

• Background

– Counterfactual Regret Minimization (CFR)

• Theoretical Advancements for CFR in:

– Many player games – Imperfect recall games

• CFR Speed-Ups
• Tricks with Memory Limitations
• Conclusion + Future Work

CFR Speed-ups

c 1 1 QJ QK 1/6 1/6 2 2 .3 .7 2 2 .3 .7 1 +1

• 2

1 1 +1 +2 .8 .2 +1

• 1

1 1 .5

• 1

+2 .5 .5

• 1
• 2

.5

L I A R ! L I A R ! P A N T S O N F I R E !

... ...

CFR Speed-ups

c 1 1 QJ QK 1/6 1/6 2 2 .3 .7 2 2 .3 .7 .5 .5 +1

• 2

.5 .5 1 +1 +2 .8 .2 +1

• 1

1 1 .5

• 1

+2 .5 .5

• 1
• 2

.5

• Each iteration, only

update action probabilities at a sampled subset of states.

... ...

CFR Speed-Ups

Chance Sampling

[Zinkevich et al., NIPS 2007]

CFR Speed-Ups

Chance Sampling External Sampling

• Faster iterations

– Use new strategies

sooner

• Need more iterations

– Good trade-off

[Zinkevich et al., NIPS 2007] [Lanctot et al., NIPS 2009]

CFR Speed-Ups

Chance Sampling External Sampling Outcome Sampling

• Even faster

iterations

• Even more

iterations required

• Good trade-
• ff?

[Zinkevich et al., NIPS 2007] [Lanctot et al., NIPS 2009]

CFR Speed-Ups

2-round Heads-Up No-Limit Hold'em, 36 chips per player

?

CFR Speed-Ups

• Outcome sampling introduces a lot of variance
• T ≤ C + K∙Variance [G. et al., AAAI 2012]

– T = Iterations required to be

“close enough” to equilibrium

– C, K = Constants

New!

[G., Lanctot, Burch, Szafron and Bowling, AAAI 2012]

CFR Speed-Ups

Chance Sampling External Sampling Outcome Sampling Average Strategy Sampling New!

Average Strategy

[G. et al., NIPS 2012] [Zinkevich et al., NIPS 2007] [Lanctot et al., NIPS 2009]

CFR Speed-Ups

2-round Heads-Up No-Limit Hold'em, 36 chips per player [G. et al., NIPS 2012]

CFR Speed-Ups

[G. et al., NIPS 2012] 2-round Heads-Up No-Limit Hold'em, k chips per player

Outline of Presentation

• Background

– Counterfactual Regret Minimization (CFR)

• Theoretical Advancements for CFR in:

– Many player games – Imperfect recall games

• CFR Speed-Ups
• Tricks with Memory Limitations
• Conclusion + Future Work

Tricks with Memory Limitations

2-player Limit Texas Hold'em Abstract Game

≈1014

≈109

≈ 59,000,000 “Turn” Deals

540,000 “Turn” Buckets

Tricks with Memory Limitations

2-player Limit Texas Hold'em Abstract Game

≈1014

≈109

3-player Limit Texas Hold'em Abstract Game

≈1017

≈109

≈ 59,000,000 “Turn” Deals

540,000 “Turn” Buckets

≈ 59,000,000 “Turn” Deals

540 “Turn” Buckets

Tricks with Memory Limitations

3-Player Limit Texas Hold'em Abstract Game Abstract Game Strategy

3-player Limit Texas Hold'em Stitched Strategy

540 “Turn” Buckets

≈ 59,000,000 “Turn” Deals

Tricks with Memory Limitations

3-Player Limit Texas Hold'em Abstract Game Abstract Game Strategy

3-player Limit Texas Hold'em Stitched Strategy

2-player Experts 2-player Sub-games

540 “Turn” Buckets

540,000 “Turn” Buckets

• Generalizes 3 previous approaches

[Gibson and Szafron, NIPS 2011]

≈ 59,000,000 “Turn” Deals

Tricks with Memory Limitations

Extensive-Form Game

Abstraction 1

Abstraction 2 Abstraction K

...

“Frankenstein” Abstract Game New!

Frankenstein-Game Strategy

Full Game Strategy

[Gibson and Szafron, NIPS 2011]

Tricks with Memory Limitations

3-player Limit Texas Hold'em

18K “Turn” Buckets

1.53 Million “Turn” Buckets

“Frankenstein” Abstract Game

Frankenstein-Game Strategy

3-player Texas Hold'em Strategy

New!

Tricks with Memory Limitations

Hyperborean3p Tournament

CFR 2-player Experts

Outline of Presentation

• Background

– Counterfactual Regret Minimization (CFR)

• Theoretical Advancements for CFR in:

– Many player games – Imperfect recall games

• CFR Speed-Ups
• Tricks with Memory Limitations
• Conclusion + Future Work

Conclusion

• We have made the following contributions:

– First set of theoretical properties for CFR in:

• games with more than 2 players
• imperfect recall games

– Theoretical and practical improvements for

making CFR go faster

– Techniques for dealing with limited memory

• This research has led to the development of

the strongest 3-player limit Texas hold'em strategies in the world.

Future Work

• Opponent modelling

– “On-line CFR”

• 10-player Texas Hold'em

– Ultimately challenge humans for

the World Series of Poker

Thanks for Listening!

• Email: richard.g.gibson@gmail.com
• Website: http://cs.ualberta.ca/~rggibson/
• Twitter: @RichardGGibson

Clip art images used in this presentation can be found at clker.com