CS 480: GAME AI MACHINE LEARNING IN GAMES 5/31/2012 Santiago Ontan - - PowerPoint PPT Presentation

CS 480: GAME AI

MACHINE LEARNING IN GAMES

5/31/2012 Santiago Ontañón santi@cs.drexel.edu https://www.cs.drexel.edu/~santi/teaching/2012/CS480/intro.html

Reminders

• Check BBVista site for the course regularly
• Also: https://www.cs.drexel.edu/~santi/teaching/2012/CS480/intro.html
• Project 4 due June 7th
• Final exam, next week: June 5th
• Tactic & Strategy
• Board Games
• Learning

Outline

• Student Presentations:
• “Game AI as Storytelling”
• “Computational Approaches to Story-telling and Creativity”
• “Procedural Level Design for Platform Games”
• Very Brief Introduction to Machine Learning
• Machine Learning in Games
• Course Summary

Outline

• Student Presentations:
• “Game AI as Storytelling”
• “Computational Approaches to Story-telling and Creativity”
• “Procedural Level Design for Platform Games”
• Very Brief Introduction to Machine Learning
• Machine Learning in Games
• Course Summary

Machine Learning

• Branch of AI that studies how to:
• Infer general knowledge from data: supervised learning
• Infer behavior from data: learning from demonstration
• Find hidden structure in data: unsupervised learning
• Infer behavior from trial an error (data): reinforcement learning
• Underlying principle is inductive inference:
• E.g. after seeing 100 times that objects fall down, we can infer by

induction the general law of gravity.

• We cannot deduce gravity from observation, we can only induce it.

Supervised Learning

• Given labeled (supervised) training data:

(x1,y1) (x2,y2) (x3,y3) … (xn,yn)

• Find the general function that maps x to y:
• f(x) = y

Examples

Supervised Learning Example

• Learning when to cross the street:
• X is defined as a list of features: (“cars coming?”, “pedestrian traffic

light color”, “time”)

• Y defined as: cross, stay
• Examples:
• x1 = (no, green, 3pm)

y1 = cross

• x2 = (no, red, 8am)

y2 = don’t cross

• x3 = (yes, red, 5pm)

y3 = don’t cross

• x4 = (yes, green, 9pm)

y4 = don’t cross

• x5 = (no, green, 10am)

y5 = cross

• Outcome of machine learning:
• f(x) = “cross if no cars coming and light color green”

Supervised Learning

• Examples can be:
• Fixed-length vector of attributes:
• Numerical
• Categorical
• Structured representations (graphs, trees, logical clauses)
• Machine learning works better with:
• Few representative attributes
• Lots of data

Simplest Method: Nearest Neighbor

• Examples are fixed vectors of real numbers
• To predict f(x):
• f(x) = yi

where (xi,yi) is the example with xi most similar to x (nearest neighbor)

Simplest Method: Nearest Neighbor

• Examples are fixed vectors of real numbers
• To predict f(x):
• f(x) = yi

where (xi,yi) is the example with xi most similar to x (nearest neighbor)

x

Simplest Method: Nearest Neighbor

• Examples are fixed vectors of real numbers
• To predict f(x):
• f(x) = yi

where (xi,yi) is the example with xi most similar to x (nearest neighbor)

x Nearest neighbor f(x) = gray

Machine Learning Methods:

• K-Nearest Neighbor
• Decision Trees (ID3, C4.5)
• Linear Regression
• Bayesian Models (Naïve Bayes)
• Boosting (AdaBoost)
• Kernel Methods (SVM)
• Neural Networks (MLP)
• Inductive Logic Programming
• etc.

Supervised Learning Methods:

• K-Nearest Neighbor
• Decision Trees (ID3, C4.5)
• Linear Regression
• Bayesian Models (Naïve Bayes)
• Boosting (AdaBoost)
• Kernel Methods (SVM)
• Neural Networks (MLP)
• Inductive Logic Programming
• etc.

Which one is the best?

Supervised Learning Methods:

• K-Nearest Neighbor
• Decision Trees (ID3, C4.5)
• Linear Regression
• Bayesian Models (Naïve Bayes)
• Boosting (AdaBoost)
• Kernel Methods (SVM)
• Neural Networks (MLP)
• Inductive Logic Programming
• etc.

Which one is the best? NONE! Each one has its own biases

Outline

• Student Presentations:
• “Game AI as Storytelling”
• “Computational Approaches to Story-telling and Creativity”
• “Procedural Level Design for Platform Games”
• Very Brief Introduction to Machine Learning
• Machine Learning in Games
• Course Summary

Why Learning in Games?

Benefit for Games:

• Adaptive, believable Game AI
• Find good player matches
• Realistic player movement
• Reduce development cost (e.g.

fine-tuning parameters)

Benefit for AI:

• Games are great test beds

(perfect instrumentation and measurements)

• Reduced cost
• Extremely challenging domains
• Great way to motivate

students!

Uses of Learning in Games

• Driving Games:
• Learn good driving behavior from observing humans
• Fine-tune parameters of physics simulation, or of car handling

parameters

• RTS Games:
• Automatically balance the game
• RPG/FPS Games:
• Believable movements
• Others:
• Specific game genres possible only with learning (training games)
• Automatically adapt to player strategies
• Learning player models

Example Usages: Black & White

Learning in Black & White

Desires Opinions Beliefs Intention: Abstract Plan Specific Plan Action List Belief-Desire-Intention Architecture

Learning in Black & White

Desires Opinions Beliefs Intention: Abstract Plan Specific Plan Action List Belief-Desire-Intention Architecture Represented as hand- crafted perceptrons (single layer neural networks) Given the current situation, they activate more or less, triggering certain desires. Example: hunger The structure of the perceptrons is hardcoded, but the parameters are learned

Learning in Black & White

Desires Opinions Beliefs Intention: Abstract Plan Specific Plan Action List Belief-Desire-Intention Architecture Represented as learned decision trees, one per desire. They capture towards which

• bject the creature should

express each desire. Example: hunger The creature will learn a decision tree from player’s feedback of what can be eaten

Learning in Black & White

Desires Opinions Beliefs Intention: Abstract Plan Specific Plan Action List Belief-Desire-Intention Architecture Beliefs are just lists of properties of the objects in the game, used for learning

• pinions.

Learning in Black & White

Desires Opinions Beliefs Intention: Abstract Plan Specific Plan Action List Belief-Desire-Intention Architecture Which desire will be attempted, and towards which object: e.g. “destroy town” or “eat human”

Learning in Black & White

Desires Opinions Beliefs Intention: Abstract Plan Specific Plan Action List Belief-Desire-Intention Architecture Desire, and list of objects that will be used. E.g. “Destroy town by throwing a stone”

Learning in Black & White

Desires Opinions Beliefs Intention: Abstract Plan Specific Plan Action List Belief-Desire-Intention Architecture Specific list of actions to execute the plan

Learning in Black & White

• BDI model to simulate character personality
• Learning is deployed at specific players to adapt the BDI

model to player preferences

• Quite advanced, for commercial game standards!

Example Usages: Forza Motorsport

Example Usages: Forza Motorsport

• Forza Motorsport lets you create “Drivatars”
• Drivatars use machine learning to “learn” to drive like the

player.

• Supervised machine learning:
• Input: the track just ahead of the car
• Output: car trajectory

Track Segments

a4 a2 a3 a1 Segments

• Define a set of “track segments”. All the tracks are made up of these
• For each segment, define the different ways in which it can be traversed

Learning from the Player

• When the player plays:
• 1. For each segment the player traverses, Drivatar

remembers the trajectory used (out of the ones it has stored), speed, and other parameters.

• 2. Drivatar only remembers the last data seen, and forgets
• ld data (to adapt to newer driving styles of the player)
• 3. When playing the game, it selects the trajectories

learned form the player for each track segment, and combines them into a minimal curvature line, which is the path taken by the AI

Minimal Curvature Line

Learning in Drivatars: Key Ideas

• Learning to drive is very complex! Too Many parameters!
• Solution: simplify the problem!
• Simplifications:
• Divide the track in segments
• Identify a finite set of possible trajectories for each segment (this

defines a discrete and finite vocabulary for the machine learning method)

• The goal of AI in games is to make the game fun, not to

create advanced AI! (at least from a game company perspective J)

Other Games Using Learning

• Colin McRae Rally 2.0: Neural networks to learn how to

drive a rally car

Other Games Using Learning

• Command & Conquer Renegade: learning new paths by
• bserving the player (e.g. player breaks a window and

creates a new path)

Other Games Using Learning

• Re-Volt: genetic algorithm tunes the parameters of the

steering behaviors for cars, to optimize lap time

Ways to Fake “Learning” in Games

• Preprogram different levels of difficulty and change

between them at run time

• Rule-based system: have a large pool of rules, and add

them bit by bit to the AI

• Change parameters during game play:
• Increase the accuracy of shooting
• Decrease reaction time of enemies
• Etc.

Outline

• Student Presentations:
• “Game AI as Storytelling”
• “Computational Approaches to Story-telling and Creativity”
• “Procedural Level Design for Platform Games”
• Very Brief Introduction to Machine Learning
• Machine Learning in Games
• Course Summary

Game AI

• Intelligent behavior for characters in games
• The goal is not to make characters smart, but to make the

game believable, engaging and fun

• There is a lot of design in game AI
• Commercial games use a collection of very simple, but

highly customizable AI techniques (FSMs, Behavior Trees, Influence Maps, etc.)

Game AI

AI World Interface (perception) Strategy Decision Making Movement

Game AI

• Perception: interfacing the AI with the game
• Movement: jumping, firing, steering behaviors
• Pathfinding: quantization, navmeshes, A*, TBA*, LRTA*, D*-Lite
• Decision Making: FSMs, Decision Trees, Behavior Trees
• Tactics/Strategy: Rule-Based systems, waypoints, influence maps
• Board Games: Minimax, alpha-beta, Monte-Carlo search
• Learning in games: supervised learning, reinforcement learning

Game AI

• Perception: interfacing the AI with the game
• Movement: jumping, firing, steering behaviors
• Pathfinding: quantization, navmeshes, A*, TBA*, LRTA*, D*-Lite
• Decision Making: FSMs, Decision Trees, Behavior Trees
• Tactics/Strategy: Rule-Based systems, waypoints, influence maps
• Board Games: Minimax, alpha-beta, Monte-Carlo search
• Learning in games: supervised learning, reinforcement learning

Projects 1, 2, 3, 4

Game AI

• Perception: interfacing the AI with the game
• Movement: jumping, firing, steering behaviors
• Pathfinding: quantization, navmeshes, A*, TBA*, LRTA*, D*-Lite
• Decision Making: FSMs, Decision Trees, Behavior Trees
• Tactics/Strategy: Rule-Based systems, waypoints, influence maps
• Board Games: Minimax, alpha-beta, Monte-Carlo search
• Learning in games: supervised learning, reinforcement learning

Midterm

Game AI

• Perception: interfacing the AI with the game
• Movement: jumping, firing, steering behaviors
• Pathfinding: quantization, navmeshes, A*, TBA*, LRTA*, D*-Lite
• Decision Making: FSMs, Decision Trees, Behavior Trees
• Tactics/Strategy: Rule-Based systems, waypoints, influence maps
• Board Games: Minimax, alpha-beta, Monte-Carlo search
• Learning in games: supervised learning, reinforcement learning

Final

Project 4

• Project 4 (and last): Rule-based Strategy for RTS Game (S3)
• Idea:
• Create a perception layer that creates a simple knowledge base (logical

terms)

• Create a simple unification algorithm with variable bindings
• Define a set of actions the rule-based system can execute
• Define a small set of rules (do not overdo it! J)
• RETE is optional (extra credit)
• See how well it plays and how easy is it to make the AI play well!
• Anyone wants to do a different project 4? Any ideas?

Next Week

• Final Exam
• Project 4 presentation