mastering the game of go with deep neural networks and
play

Mastering the game of Go with deep neural networks and tree search - PowerPoint PPT Presentation

Mar 01, 2024 •233 likes •907 views

David Silver et al. from Google DeepMind Mastering the game of Go with deep neural networks and tree search Article overview by Reinforcement Learning Seminar Ilya Kuzovkin University of Tartu, 2016 T HE G AME OF G O B OARD B OARD S TONES B

  1. David Silver et al. from Google DeepMind Mastering the game of Go with deep neural networks and tree search Article overview by Reinforcement Learning Seminar Ilya Kuzovkin University of Tartu, 2016

  2. T HE G AME OF G O

  3. B OARD

  4. B OARD S TONES

  5. B OARD S TONES G ROUPS

  6. B OARD L IBERTIES S TONES G ROUPS

  7. B OARD L IBERTIES C APTURE S TONES G ROUPS

  8. B OARD L IBERTIES C APTURE K O S TONES G ROUPS

  9. B OARD L IBERTIES E XAMPLES C APTURE K O S TONES G ROUPS

  10. B OARD L IBERTIES E XAMPLES C APTURE K O S TONES G ROUPS

  11. B OARD L IBERTIES E XAMPLES C APTURE K O S TONES G ROUPS

  12. B OARD L IBERTIES E XAMPLES C APTURE K O S TONES G ROUPS T WO EYES

  13. F INAL COUNT B OARD L IBERTIES E XAMPLES C APTURE K O S TONES G ROUPS T WO EYES

  14. T RAINING

  15. T RAINING THE BUILDING BLOCKS S UPERVISED S UPERVISED R EINFORCEMENT CLASSIFICATION REGRESSION Supervised Reinforcement Value � policy network policy network network p σ ( a | s ) p ρ ( a | s ) v θ ( s ) Rollout policy network p π ( a | s ) Tree policy network p τ ( a | s )

  16. Supervised policy network p σ ( a | s )

  17. Supervised policy network p σ ( a | s ) Softmax 1 convolutional layer 1x1 ReLU 11 convolutional layers 3x3 with k =192 filters, ReLU 1 convolutional layer 5x5 with k =192 filters, ReLU 19 x 19 x 48 input

  18. • 29.4M positions from games Supervised between 6 to 9 dan players policy network p σ ( a | s ) Softmax 1 convolutional layer 1x1 ReLU 11 convolutional layers 3x3 with k =192 filters, ReLU 1 convolutional layer 5x5 with k =192 filters, ReLU 19 x 19 x 48 input

  19. • 29.4M positions from games Supervised between 6 to 9 dan players policy network p σ ( a | s ) Softmax • stochastic gradient ascent � • learning rate = 0.003, 1 convolutional layer 1x1 α ReLU halved every 80M steps � 11 convolutional layers 3x3 • batch size m = 16 with k =192 filters, ReLU � • 3 weeks on 50 GPUs to make 1 convolutional layer 5x5 340M steps with k =192 filters, ReLU 19 x 19 x 48 input

  20. • 29.4M positions from games Supervised between 6 to 9 dan players policy network • Augmented: 8 reflections/rotations • Test set (1M) accuracy: 57.0% p σ ( a | s ) • 3 ms to select an action Softmax • stochastic gradient ascent � • learning rate = 0.003, 1 convolutional layer 1x1 α ReLU halved every 80M steps � 11 convolutional layers 3x3 • batch size m = 16 with k =192 filters, ReLU � • 3 weeks on 50 GPUs to make 1 convolutional layer 5x5 340M steps with k =192 filters, ReLU 19 x 19 x 48 input

  21. 19 X 19 X 48 INPUT

  22. 19 X 19 X 48 INPUT

  23. 19 X 19 X 48 INPUT

  24. 19 X 19 X 48 INPUT

  25. Rollout policy p π ( a | s ) • Supervised — same data as p σ ( a | s ) • Less accurate: 24.2% ( vs. 57.0% ) • Faster: 2 μ s per action ( 1500 times ) • Just a linear model with softmax

  26. Rollout policy p π ( a | s ) • Supervised — same data as p σ ( a | s ) • Less accurate: 24.2% ( vs. 57.0% ) • Faster: 2 μ s per action ( 1500 times ) • Just a linear model with softmax

  27. Rollout policy p π ( a | s ) Tree policy p τ ( a | s ) • Supervised — same data as p σ ( a | s ) • Less accurate: 24.2% ( vs. 57.0% ) • Faster: 2 μ s per action ( 1500 times ) • Just a linear model with softmax

  28. Rollout policy p π ( a | s ) Tree policy p τ ( a | s ) • Supervised — same data as • “ similar to the rollout policy but p σ ( a | s ) • Less accurate: 24.2% ( vs. 57.0% ) with more features ” • Faster: 2 μ s per action ( 1500 times ) • Just a linear model with softmax

  29. Reinforcement policy network p ρ ( a | s ) Same architecture Weights are initialized with ρ σ

  30. • Self-play: current network vs. Reinforcement randomized pool of previous versions policy network p ρ ( a | s ) Same architecture Weights are initialized with ρ σ

  31. • Self-play: current network vs. Reinforcement randomized pool of previous versions policy network p ρ ( a | s ) • Play a game until the end, get the reward z t = ± r ( s T ) = ± 1 Same architecture Weights are initialized with ρ σ

  32. • Self-play: current network vs. Reinforcement randomized pool of previous versions policy network p ρ ( a | s ) z t = ± r ( s T ) = ± 1 • Play a game until the end, get the reward z i • Set and play the same game again, this time t = z t updating the network parameters at each time step t Same architecture Weights are initialized with ρ σ

  33. • Self-play: current network vs. Reinforcement randomized pool of previous versions policy network p ρ ( a | s ) z t = ± r ( s T ) = ± 1 • Play a game until the end, get the reward z i • Set and play the same game again, this time t = z t updating the network parameters at each time step t v ( s i t ) • = … ‣ 0 “ on the first pass through the training pipeline ” v θ ( s i t ) ‣ “ on the second pass ” Same architecture Weights are initialized with ρ σ

  34. • Self-play: current network vs. Reinforcement randomized pool of previous versions policy network p ρ ( a | s ) z t = ± r ( s T ) = ± 1 • Play a game until the end, get the reward z i • Set and play the same game again, this time t = z t updating the network parameters at each time step t v ( s i t ) • = … ‣ 0 “ on the first pass through the training pipeline ” v θ ( s i t ) ‣ “ on the second pass ” • batch size n = 128 games • 10,000 batches • One day on 50 GPUs Same architecture Weights are initialized with ρ σ

  35. • Self-play: current network vs. Reinforcement randomized pool of previous versions policy network • 80% wins against Supervised Network • 85% wins against Pachi (no search yet!) p ρ ( a | s ) • 3 ms to select an action z t = ± r ( s T ) = ± 1 • Play a game until the end, get the reward z i • Set and play the same game again, this time t = z t updating the network parameters at each time step t v ( s i t ) • = … ‣ 0 “ on the first pass through the training pipeline ” v θ ( s i t ) ‣ “ on the second pass ” • batch size n = 128 games • 10,000 batches • One day on 50 GPUs Same architecture Weights are initialized with ρ σ

  36. Value � network v θ ( s )

  37. Value � network v θ ( s ) Fully connected layer 1 tanh unit Fully connected layer 256 ReLU units 1 convolutional layer 1x1 ReLU 11 convolutional layers 3x3 with k =192 filters, ReLU 1 convolutional layer 5x5 with k =192 filters, ReLU 19 x 19 x 49 input

  38. Evaluate the value of the position s under policy p: • Value � � network Double approximation • v θ ( s ) Fully connected layer 1 tanh unit Fully connected layer 256 ReLU units 1 convolutional layer 1x1 ReLU 11 convolutional layers 3x3 with k =192 filters, ReLU 1 convolutional layer 5x5 with k =192 filters, ReLU 19 x 19 x 49 input

  39. Evaluate the value of the position s under policy p: • Value � � network Double approximation • v θ ( s ) • Stochastic gradient descent to Fully connected layer minimize MSE 1 tanh unit Fully connected layer 256 ReLU units 1 convolutional layer 1x1 ReLU 11 convolutional layers 3x3 with k =192 filters, ReLU 1 convolutional layer 5x5 with k =192 filters, ReLU 19 x 19 x 49 input

  40. Evaluate the value of the position s under policy p: • Value � � network Double approximation • v θ ( s ) • Stochastic gradient descent to Fully connected layer minimize MSE 1 tanh unit • Train on 30M state-outcome Fully connected layer ( s, z ) pairs , each from a unique 256 ReLU units game generated by self-play: 1 convolutional layer 1x1 ReLU 11 convolutional layers 3x3 with k =192 filters, ReLU 1 convolutional layer 5x5 with k =192 filters, ReLU 19 x 19 x 49 input

  41. Evaluate the value of the position s under policy p: • Value � � network Double approximation • v θ ( s ) • Stochastic gradient descent to Fully connected layer minimize MSE 1 tanh unit • Train on 30M state-outcome Fully connected layer ( s, z ) pairs , each from a unique 256 ReLU units game generated by self-play: ‣ choose a random time step u 1 convolutional layer 1x1 ‣ sample moves t =1… u -1 from ReLU SL policy ‣ make random move u 11 convolutional layers 3x3 ‣ sample t = u +1…T from RL with k =192 filters, ReLU policy and get game 1 convolutional layer 5x5 outcome z with k =192 filters, ReLU ( s u , z u ) ‣ add pair to the training set 19 x 19 x 49 input

  42. Evaluate the value of the position s under policy p: • Value � � network Double approximation • v θ ( s ) • Stochastic gradient descent to Fully connected layer minimize MSE 1 tanh unit • Train on 30M state-outcome Fully connected layer ( s, z ) pairs , each from a unique 256 ReLU units game generated by self-play: ‣ choose a random time step u 1 convolutional layer 1x1 ‣ sample moves t =1… u -1 from ReLU SL policy ‣ make random move u 11 convolutional layers 3x3 ‣ sample t = u +1…T from RL with k =192 filters, ReLU policy and get game 1 convolutional layer 5x5 outcome z with k =192 filters, ReLU ( s u , z u ) ‣ add pair to the training set 19 x 19 x 49 input • One week on 50 GPUs to train on 50M batches of size m =32

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Mastering the game of Go with deep neural networks and tree search Nature, Jan, 2016 Roadmap

Mastering the game of Go with deep neural networks and tree search Nature, Jan, 2016 Roadmap

Mastering the game of Go with deep neural networks and tree search Nature, Jan, 2016 Roadmap What this paper is about? Deep Learning Search problem How to explore a huge tree (graph) AlphaGo Video

556 views • 51 slides
Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree Search, Nature 2016] CS 486/686 University of Waterloo Lecture 21: July 12, 2017 Outline AlphaGo Supervised Learning of Policy Networks

540 views • 15 slides
Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan Zelener Machine Learning Reading Group January 7 th 2016 The Graduate Center, CUNY Objectives Explain some of the trends of deep learning and

680 views • 44 slides
Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking

Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking

Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In

987 views • 40 slides
Visualizing and Interpreting Deep Neural Networks Bolei Zhou Department of Information

Visualizing and Interpreting Deep Neural Networks Bolei Zhou Department of Information

Visualizing and Interpreting Deep Neural Networks Bolei Zhou Department of Information Engineering The Chinese University of Hong Kong Deep Neural Networks are Everywhere Playing Go Making Medical Decision Understanding Scenes Deep Neural

1.1k views • 46 slides
On the Expressive Power of Deep Neural Networks Maithra Raghu, Ben Poole, Jon Kleinberg, Surya

On the Expressive Power of Deep Neural Networks Maithra Raghu, Ben Poole, Jon Kleinberg, Surya

On the Expressive Power of Deep Neural Networks Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, Jascha Sohl Dickstein Tom Brady Deep Neural Networks Recent successes in using Deep neural networks for image classification,

1.32k views • 17 slides
Outline Evolution of neurocomputing Artificial neural networks Feed forward

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Introduction to Neural Networks and Deep Learning Ling Guan References: S. Haykin, Neural Networks , 2 nd Edition, New Jersey: Prentice Hall, 1. 2004. C. M. Bishop, Neural Networks for Pattern Re cognition, New York: 2. Oxford University

451 views • 17 slides
Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova

Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova

Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova November 21, 2016 Neural Networks 2/20 Neural Networks 3/20 Neural Networks Neural computing requires a number of neurons , to be connected

813 views • 21 slides
Optimizing Deep Neural Networks Leena Chennuru Vankadara 26-10-2015 Table of Contents Neural

Optimizing Deep Neural Networks Leena Chennuru Vankadara 26-10-2015 Table of Contents Neural

Optimizing Deep Neural Networks Leena Chennuru Vankadara 26-10-2015 Table of Contents Neural Networks and loss surfaces Problems of Deep Architectures Optimization in Neural Networks Under fitting Proliferation of Saddle

1.67k views • 41 slides
Deep Neural Networks and Molecular Dynamics Roberto Car Princeton University MAX Conference

Deep Neural Networks and Molecular Dynamics Roberto Car Princeton University MAX Conference

Deep Neural Networks and Molecular Dynamics Roberto Car Princeton University MAX Conference ICTP, Trieste, Jan 29-31 2018 Deep Neural Networks When properly trained, deep NN are capable of learning the complex functional dependence of

442 views • 17 slides
CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided

CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided

CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided Backpropagation, Deep Dream, Deep Art, Fooling Convolutional Neural Networks Mitesh M. Khapra Department of Computer Science and Engineering Indian Institute

1.07k views • 72 slides
Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and Courville [chapt. 6,7,8]; AIMA [sect. 21.1-21.3]; Sutton and Barto, Reinforcement Learning: an

527 views • 35 slides
Propagating Error Backward Hyperparameters for Neural Networks } Multi-layer (deep) neural

Propagating Error Backward Hyperparameters for Neural Networks } Multi-layer (deep) neural

Learning in Neural Networks w 1,3 w 3,5 1 3 5 w w 1,4 3,6 w w 2,3 4,5 2 4 6 w w 2,4 4,6 Class #18: Back-Propagation; } A neural network can learn a classification function by Tuning Hyper-Parameters adjusting its weights to

259 views • 4 slides
Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the

Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the

Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Ventral Stream Umut Gcl and Marcel A. J. van Gerven Article overview by Ilya Kuzovkin Computational Neuroscience Seminar University of Tartu

852 views • 63 slides
Recurrent Neural Networks Deep neural networks have enabled major advances in machine learning

Recurrent Neural Networks Deep neural networks have enabled major advances in machine learning

Recurrent Neural Networks Deep neural networks have enabled major advances in machine learning and AI y t-1 y t y t+1 Computer vision h t-1 h t h t+1 Language translation Speech recognition h t-1 h t h t+1 Question answering x t-1 x t x t+1

454 views • 43 slides
Concise Introduction to Deep Neural Networks Outline: Classification problems Motivating

Concise Introduction to Deep Neural Networks Outline: Classification problems Motivating

Concise Introduction to Deep Neural Networks Outline: Classification problems Motivating Deep (large) Neural Network (DNN) classifiers Neurons and DNN architectures Numerical training of DNNs (supervised deep learning) Spiking

630 views • 33 slides
@ Mastering the Millennial Mindset and Beyond How to Attract and Retain Emerging Leaders Lisa

@ Mastering the Millennial Mindset and Beyond How to Attract and Retain Emerging Leaders Lisa

@ Mastering the Millennial Mindset and Beyond How to Attract and Retain Emerging Leaders Lisa Ryan Cleveland Where are you from originally? Would you be surprised to know Social Dis istancing Who said it? (chat box guesses) The

443 views • 28 slides
Effect of Non-Passive Operator on Enhanced Wave-Based Teleoperator for Robotic-Assisted Surgery:

Effect of Non-Passive Operator on Enhanced Wave-Based Teleoperator for Robotic-Assisted Surgery:

Motivation Enhanced Wave Variable Architecture Conclusions and Perspectives Effect of Non-Passive Operator on Enhanced Wave-Based Teleoperator for Robotic-Assisted Surgery: First Case Study Jing Guo*, Chao Liu, Philippe Poignet Department of

707 views • 28 slides
Sector/Sphere Tutorial Yunhong Gu CloudCom 2010, Nov. 30, Indianapolis, IN Outline Outline

Sector/Sphere Tutorial Yunhong Gu CloudCom 2010, Nov. 30, Indianapolis, IN Outline Outline

Sector/Sphere Tutorial Yunhong Gu CloudCom 2010, Nov. 30, Indianapolis, IN Outline Outline Introduction to Sector/Sphere Major Features Installation and Configuration Use Cases The Sector/Sphere Software The Sector/Sphere

471 views • 46 slides
Invasive Malleable Applications Sebastian Buchwald, Manuel Mohr, Andreas Zwinkau Karlsruhe

Invasive Malleable Applications Sebastian Buchwald, Manuel Mohr, Andreas Zwinkau Karlsruhe

Invasive Malleable Applications Sebastian Buchwald, Manuel Mohr, Andreas Zwinkau Karlsruhe Institute of Technology TCRC 89 "Invasive Computing" ATPS 2015 No faster CPUs, only more of them More cores means slower cores M. B. Taylor,

531 views • 25 slides
[Transition from Matts presentation] Before the University Libraries at UNCG began making the

[Transition from Matts presentation] Before the University Libraries at UNCG began making the

[Transition from Matts presentation] Before the University Libraries at UNCG began making the transition to WorldShare Management Systems, I had only been through one data migrationand that was years ago from the safe distance of a student

700 views • 16 slides
Mastering the Game of Go without Human Knowledge 06/15/18 Presented by: Henry Chen CS885

Mastering the Game of Go without Human Knowledge 06/15/18 Presented by: Henry Chen CS885

Mastering the Game of Go without Human Knowledge 06/15/18 Presented by: Henry Chen CS885 Reinforcement Learning Introduction Image source: https://medium.com/syncedreview/alphago-zero-approaching-perfection-d8170e2b4e48 PAGE 2 Introduction

455 views • 22 slides
DATABASE SYSTEM IMPLEMENTATION GT 4420/6422 // SPRING 2019 // @JOY_ARULRAJ LECTURE #8: DATABASES

DATABASE SYSTEM IMPLEMENTATION GT 4420/6422 // SPRING 2019 // @JOY_ARULRAJ LECTURE #8: DATABASES

DATABASE SYSTEM IMPLEMENTATION GT 4420/6422 // SPRING 2019 // @JOY_ARULRAJ LECTURE #8: DATABASES ON NEW HARDWARE 2 ANATOMY OF A DATABASE SYSTEM Process Manager Connection Manager + Admission Control Query Parser Query Processor Query

1.15k views • 89 slides
ADEPTSHIP AD MYSTIC ADEPT We are ready to transfer to you blessings and initiations and an

ADEPTSHIP AD MYSTIC ADEPT We are ready to transfer to you blessings and initiations and an

ADEPTSHIP AD MYSTIC ADEPT We are ready to transfer to you blessings and initiations and an impetus for greater life within your temple. I ask you, then, to study all that you know about what the adepts have taught, to do some form of

543 views • 15 slides

More recommend