SLIDE 1
Deep Learning for Video Game Playing
Authors: Niels Justesen, Philip Bontrager, Julian Togelius, Sebastian Risi
Presented by: Runsheng (Benson) Guo
1
Outline
- Background
- Methods
- History
- Open Challenges
- Recent Advances
2
Deep Learning for Video Game Playing Authors: Niels Justesen, Philip - - PowerPoint PPT Presentation
Deep Learning for Video Game Playing Authors: Niels Justesen, Philip - - PowerPoint PPT Presentation
Deep Learning for Video Game Playing Authors: Niels Justesen, Philip Bontrager, Julian Togelius, Sebastian Risi Presented by: Runsheng (Benson) Guo 1 Outline Background Methods History Open Challenges Recent Advances 2
SLIDE 2
SLIDE 3
Background: Neural Networks
Convolutional Neural Network Recurrent Neural Network
3
SLIDE 4
- Supervised Learning
Background: Neural Network Optimization
- Unsupervised Learning
- Reinforcement Learning
- Evolutionary Approaches
- Hybrid Learning Approaches
4
SLIDE 5
Methods
Genres:
- Arcade Games
- Racing Games
- First-Person Shooters
- Open-World Games
- Real-Time Strategy
- Text Adventure Games
Platforms:
- Arcade Learning Environment (ALE)
- Retro Learning Environment (RLE)
- OpenAI Gym
- Many more!
5
SLIDE 6
Methods: Arcade Games
Characteristics:
- 2-Dimensional Movement
- Continuous-time Actions
Challenges:
- Precise timing
- Environment navigation
- Long term planning
6
SLIDE 7
Methods: Arcade Games
Deep Q-Learning:
- Replay buffer, separate target network, recurrent layer
- Distributed DQN
- Double DQN
- Prioritized experience replay
- Dueling DQN
- NoisyNet DQN
- Rainbow
Actor-Critic:
- A3C
- IMPALA
- UNREAL
7
SLIDE 8
Methods: Arcade Games
Other Algorithms:
- Deep GA
- Frame prediction
- Hybrid reward architecture
Montezuma’s Revenge:
- Very sparse rewards
- Hierarchical DQN
- Density models
- Text instructions
8
SLIDE 9
Methods: Racing Games
Characteristics:
- Minimize navigation time
- Continuous-time Actions
Challenges:
- Precise inputs
- Short & long term planning
- Adversarial planning
9
SLIDE 10
Methods: Racing Games
Paradigms:
- Behaviour reflex (sensors → action)
- Direct perception (sensors → environment information → action)
Algorithms:
- (Deep) Deterministic policy gradient
- A3C
10
SLIDE 11
Methods: First-Person Shooters
Characteristics:
- 3-Dimensional Movement
- Player Interaction
Challenges:
- Fast reactions
- Predicting enemy actions
- Teamwork
11
SLIDE 12
Methods: First-Person Shooters
Algorithms:
- Deep Q-learning
- A3C
○ UNREAL ○ Reward shaping ○ Curriculum learning
- Direct future prediction
- Distill and transfer learning
- Intrinsic curiosity module
12
SLIDE 13
Methods: Open-World Games
Characteristics:
- Large world to explore
- No clear goals
Challenges:
- Setting meaningful goals
- Large action space
13
SLIDE 14
Methods: Open-World Games
Algorithms:
- Hierarchical deep reinforcement learning network
- Teacher-student curriculum learning
- Neural turing machines
○ Recurrent memory Q-network ○ Feedback recurrent memory Q-network
14
SLIDE 15
Methods: Real-Time Strategy
Characteristics:
- Control multiple units
simultaneously
- Continuous-time Actions
Challenges:
- Long term planning
- Delayed rewards
15
SLIDE 16
Methods: Real-Time Strategy
Algorithms:
- Unit control
○ Zero order optimization ○ Independent Q-learning ○ A3C ■ Multiagent Bidirectionally-Coordinated Network ■ Counterfactual Multi-Agent
- Build order planning
○ Supervised learning ○ Reinforcement learning ■ Double DQN ■ Proximal Policy Optimization
16
SLIDE 17
Methods: Text Adventure Games
Characteristics:
- Text-only states & actions
- Choice, hyperlink & parser
interfaces Challenges:
- Natural language processing
- Large action space
17
SLIDE 18
Methods: Text Adventure Games
Algorithms:
- LSTM-DQN
- Deep Reinforcement Relevance Net
- State affordances
- Action elimination network
18
SLIDE 19
History
Trends:
- Incremental extensions
○ DQN ○ A3C
- Parallelization
○ A3C ○ Evolutionary algorithms
19
SLIDE 20
Open Challenges
- Agent modelling
○ General game playing ○ Human-like behaviour ○ Delayed/sparse rewards, multi-agent learning, dealing with large action spaces
- Game industry Adoption
- Developing model-based algorithms
- Improving computational efficiency
20
SLIDE 21
Conclusion
Recent Advances:
- Model-Based Reinforcement Learning for Atari (Kaiser et al, 2019)
- AlphaStar (DeepMind, 2019)
- OpenAI Five (OpenAI, 2019)
Future Work:
- Survey focusing on a single class of deep learning algorithms
- Survey focusing on a single genre of video games
21