FIT CTU

Adam Vesecký

vesecky.adam@gmail.com

Lecture 10

Game AI

Architecture of Computer Games

Adam Vesecký

AI introduction

Game AI

Game AI has something to do with scripting, a lot to do with state machines, even more to do with map analysis, and not a lot to do with AI field.

Artificial Intelligence in Games

  • a broad set of principles that generate behaviors, providing an illusion of intelligence
  • should be sophisticated enough to make the players think they interact with something intelligent
  • even though it’s interesting when AI does unpredictable things, it’s not necessarily fun for players
AI in games
AI in games
AI in games

AI Techniques

AI techniques

AI Techniques

AI flow

AI in games

Doom 1 (1993)

  • monsters infighting (two AI characters encounter each other)
  • state machines, intelligent map (SPECIAL attribute of map lines)

Age of Empires (1997)

  • AI is smart enough to find and revisit other kingdoms, poor in self-defense

Half-life (1998)

  • monsters can hear and track players, flee when getting defeated etc.
  • soldiers had automated announcement system, ability to cover

Starcraft (1998)

  • particle model for hidden units' position estimation

Unreal Tournament (1999)

  • Bayesian networks

Quake III (1999)

  • HFSM for decisions, AAS (Area Awareness System) for collisions and pathfinding
Gordon Freeman

AI techniques in games

Halo 2 (2004)

  • behavior trees and fuzzy logic

F.E.A.R. (2005)

  • GOAP for tactical coordinating with squad members, suppression fire, blind fire

XCOM: Enemy Unknown (2012)

  • utility-based system, measuring usefulness for every possible action

ARMA 3 (2013)

  • efficient issue ordering, situation-based decisions, retreats, ambush
  • HTN (Hierarchical task network) for mission generator

The Last Of Us (2013)

  • Ellie - companion AI

DOTA 2 (2013)

  • OpenAI bot (since 2017) has beaten some of the greatest players

Starcraft II (2013)

  • AlphaStar deep neural network, released in 2019
Starcraft

Challenges for Game AI

Game AI features and limits

  • real-time
  • limited resources
  • incomplete knowledge
  • planning
  • learning

Game AI properties

  • predictability and unpredictability (surprise elements)
  • support - communication between NPC and the player
  • surprise - harassment, ambush, team support,...
  • winning well and losing well
  • cheating - acceptable as long as it doesn't get detected by the player
AI challenges
The AI must be fun to play against, not beat the player easily

AI for mobs

Scripting

  • IF-THIS-THEN-THAT
  • AI behavior is completely hardcoded
  • simple, easy to debug, easy to extend
  • human player should behave as the developers expected
  • good scripting behavior must cover a large amount of situations
1 // Doom 2: find player to chase
2 void A_Look (mobj_t* actor) {
3 mobj_t* targ;
4 actor->threshold = 0; // any shot will wake up
5 targ = actor->subsector->sector->soundtarget;
6 if (actor->flags & MF_AMBUSH){
7 if (P_CheckSight (actor, actor->target))
8 goto seeyou;
9 } else goto seeyou;
10
11 if (!P_LookForPlayers (actor, false)) return;
12 // go into chase state
13 seeyou:
14 P_ChasePlayer();
15 }
Doom scripting

Finite State Machine

  • the oldest and most commonly used formalism to model game AIs
  • useful for entities with a small set of distinct states
    Dungeon FSM
  • each entity can be in exactly one of a finite number of states at any time
  • Definition
    • quadruple:
    •  is a finite, non-empty set of states
    •  is a finite set of inputs
    •  is the state-transition function
    •  is an initial state,
  • can be implemented via polymorphism or a state transition table
  • unmanageable for large complex systems, leading to transition explosion

Example: Pacman FSM

FSM pacman
pacman

Example: Pacman transition table

StateTransitionCondition
Wander the mazeChase pacmanPacman spotted
Wander the mazeFlee PacmanPowerPellet eaten
Chase PacmanWander the mazePacman lost
Chase PacmanFlee PacmanPowerPellet eaten
Flee PacmanReturn to BaseEaten by Pacman
Flee PacmanWander the mazePowerPellet expired
Return to BaseWander the mazeCentral base reached

Example: Doomguard

FSM doomguard
doomguard

Example: Doomguard

  • let's add an ability to fall asleep
doomguard
FSM doomguard extended

Hierarchical state machine

  • also known as statecharts
  • each state can have a superstate or a set of substates
  • groups of states share transitions
  • usually implemented as a stack
    • push a low-level state on the stack when entered
    • pop and move to the next state when finished
HFSM doomguard
doomguard

Fuzzy logic

  • a complementary asset for state machines and scripting
  • instead of thresholding, we can blurry out state transitions
  • used in Halo 3 for unit control, threat assessment and classification
fuzzy logic diagram

Behavior Tree

  • tree of hierarchical nodes that control decision making process
  • originated from gaming industry since Halo 2 (2004)
  • combines elements from both Scripting and HFSMs
  • there is no standardized formalization
  • depth-first traversal, starting with the root node
  • each executed behavior passes back and returns a status
    • SUCCESS, FAILURE, RUNNING, (SUSPENDED)
behavior tree example

Behavior Tree

behavior tree
Node TypeSuccessFailureRunning
SelectorIf one child succeedsIf all children failIf one child is running
SequenceIf all children succeedIf one child failsIf one child is running
DecoratorIt dependsIt dependsIt depends
ParallelIf N children succeedIf M-N children succeedIf all children are running
ActionWhen completedUpon an errorDuring completion
ConditionIf trueIf falseNever

Behavior Tree - Node Types

Selector

  • tries to execute its child nodes from left to right until it receives a successful response
  • when a successful response is received, responds with a success

Sequence

  • will execute each of its child nodes in a sequence from left to right until a failure is received
  • if every child node responds with a success, the sequence node itself will respond with a success

Decorator

  • allows for additional behavior to be added without modifying the code
  • only one child node
  • returns success if the specified condition is met, and its subtree has been executed successfully
  • Example: Repeater, Invertor, Succeeder, Failer, UntilFail, UntilSucceed

Behavior Tree - Node Types

Parallel node

  • a node that concurrently executes all of its children

Condition

  • a node that will observe the state of the game environment and respond with either a success or a failure based on the observation, condition etc.
  • Instant Check condition - the check is run once
  • Monitoring condition - keeps checking a condition over time

Action

  • a node used to interact with the game environment
  • may represent an atomic action, a script or even another BT

Blackboard

  • a construct that stores whatever information is written to it
  • any node inside or outside the tree has an access to the blackboard

Example: Unreal Engine BT Editor

behavior tree in unrel engine

Example: Doomguard

behavior tree with doomguard

BT Improvements

  • we can define a conditional selector in order to simplify the diagram
behavior tree optimizationbehavior tree optimization

Example: Locked door

doom door opening
behavior tree door

Goal-Oriented Action Planning

  • centers on the idea of goals as desirable world states -> select a goal and attempt to fulfill it
  • each action has a set of conditions it can satisfy, as well as a set of preconditions that must be true in order to be satisfied
  • implemented for F.E.A.R (2005) and Tomb Raider (2013)
goal-oriented action planning diagram

Example: F.E.A.R.

  • Advanced AI behavior: blind fire, dive through windows, take cover,...
  • key idea: to determine when to switch and what parameters to set
F.E.A.R.

Example: F.E.A.R

  • a set of goals is assigned to each mob
  • these goals compete for activation, and the AI uses a planner to try to satisfy the highest priority goal
  • the AI figures out the dependencies at run-time, based on the goal state and effects of actions
SoldierAssassinRat
AttackAttackAnimate
AttackCrouchInspectDisturbanceIdle
SuppressionFireLookAtDisturbanceGotoNode
FlushOutWithGrenadeAttackMeleeUncloaked
AttackFromCoverTraverseBlockedDoor
BlindFireFromCoverAttackFromAmbush
ReloadCoveredAttackLungeUncloaked
FEAR soldier
FEAR assassin
FEAR rat

AI for navigation

Predicting opponents

  • cheating bots are not fun in general
  • bots shouldn't know where the player is
    predicting opponents

Partial observations of the game state

  • AI is informed in the same way as the human player
  • the mistakes these models make are more human-like
  • Filtering problem - estimating the internal states based on partial observations
  • General methods:
    • Hidden semi-markov model
    • Particle filters
  • Methods for FPS:
    • Occupancy map
    • SOAR cognitive architecture (used in Quake II)
    • Vision Cone (used in Bioshock)
  • Methods for strategies:
    • Threat map
    • Influence map
    • Bayesian networks

Occupancy map

  • a grid over the game environment
  • maintains probability for each grid cell
  • probability represents the likelihood of an opponent presence
  • when the opponent is not visible, the probabilities from the previous occupancy cells are propagated along the edges to neighboring cells
occupancy map animation

Threat map

  • the value at any coordinate is the damage the enemy nearby can inflict
  • can be easily integrated into pathfinding
  • Example: combat units bypass enemy defensive structure in order to attack their infrastructure
threat map diagram

Example: OpenTTD

  • business simulator, derived from Transport Tycoon Deluxe
  • many challenges for AI
    • management over particular transport types
    • financial issues handling (loan control)
    • maintenance and vehicle replacement
    • acceleration model
    • upgrades, road extending
    • pathfinding, partial reusability of existing roads
    • vehicle utilization, failures, traffic
Transport Tycoon

OpenTTD AI

PathZilla AI

  • advanced graph theory
  • Delaunay Triangulation
  • roads based on minimum spanning tree
  • queue of planned actions

trAIns

  • most scientific approach, evolved from a research project in 2008
  • double railways support
  • concentration on production

AdmiralAI

  • attempt to implement as many features from game API as possible
  • the oldest and most comprehensive AI for OpenTTD

NoCAB

  • very fast network development and company value grow
  • custom A* implementation designed for speed
  • can upgrade vehicles

Example: Pathzilla Road Analysis

Pathzilla

Pathzilla

Triangulation

Pathzilla

Min spanning tree

Pathzilla

Shortest path tree

Pathzilla

Planning graph

Pathzilla

Pathzilla

Pathzilla

Final road network

Real AI in games

Terms

Planning

  • a formalized process of searching for sequence of actions to satisfy a goal

Supervised learning

  • works just like learning at schools
  • for certain input, there is a correct output the algorithm has to learn

Unsupervised learning

  • the output cannot be categorized as being either correct or false
  • the algorithm learns patterns instead

Reinforcement learning

  • an agent must learn the best possible output by using trial and error
  • for each action chosen the agent obtains a feedback

Reinforcement learning

  • learning what to do and how to map states to action to maximize a reward
  • based on its action, the agent gets a reward or punishment - these rewards are used to learn
  • Reward - informs the agent to which outcome the selected action leads
  • Exploration - probing new regions of the search space to find new promising solutions
  • Exploitation - probing promising region of the search space to improve current solution

Algorithms

  • Genetic Evolution
  • Q-Learning
  • SARSA
  • Monte-Carlo simulation

Models

  • Decision trees
  • Bayesian networks
  • Neural networks
MonteCarlo tree search

Monte-Carlo tree search

Artificial neural network

  • computational model that attempts to simulate a biological neural network
  • network of nodes are connected by links that have numeric weights attached
  • backpropagation - measure of divergence between the observed and the desired output
Neural network diagram
dragon
fireball
mage
platform

Neural Network Applications

MarI/O

  • NEAT - genetic algorithm for evolving neural networks (LINK)
  • Input: simplified game screen (13x13) blocks
  • Output: pressed button (6 neurons)
  • Max neurons: 1 000 000
MarI/O neural network
MARI/O

DeepMind 2014

  • neural network that can play Atari games (uses Q-learning)
  • Inputs: 210x160 at 60Hz, terminal signal and game score
DeepMind on Atari

Case-based reasoning

Main idea

  • when making decisions, we remember previous experiences and then apply solutions that worked in previous situations
  • if a certain situation hasn't been encountered yet, we recall similar experience and adjust the solution to fit the current situation

Steps

  • Retrieve - get the cases which are relevant to the given problem
  • Reuse - create a preliminary solution from selected cases
  • Revise - adapt the preliminary solution to fit the current situation
  • Retain - save the case into the knowledge base

Example: Defcon

  • multiplayer real-time strategy game
  • players compete in a nuclear war to score points by hitting opponent cities
  • stages: placing resources, fleet maneuvres, fighter attacks, final missile strikes
  • AI: decision tree learning and case-based reasoning
DEFCON

Defcon: Case structure

  • starting positions of units and buildings
  • metafleet movement and attack plan
  • performance statistics for deployed resources
  • abstraction of the opponent attacks (time-framed clustering)
  • routes taken by opponent fleets
  • final scores of the two players
DEFCON Case-Based diagram

AI in strategies

Real-time strategy

  • Real-time strategy is a Bayesian, zero-sum game (Rubinstein, 1994)
    Total War
  • a game where the player is in control of certain, usually military, assets, with which the player can manipulate in order to achieve victory
  • goal: build up a base, gather resources, produce army, destroy the enemy
  • methods: layer-based AI, rule-based AI

Main elements

  • map, mini-map
  • resources
  • units and their attributes
  • buildings

Other features

  • real-time aspect (no turns)
  • fog of war
  • tech tree

RTS games

Dune II
1992
C&C Red Alert
1995
Total Annihilation
1997
Warcraft 3: Reign of Chaos
2002
Starcraft II
2010
Ashes of the Singularity
2016

RTS Features

Resource Control

  • minerals, gas, oil, trees,...
  • controlling more resources increases the players' constsruction capabilities

Tech tree

  • a directed acyclic graph that contains the whole technological development of a faction

Build order (opening)

  • the timing at which the first buildings are constructed
RTS build order

Fog of war

  • fog that covers the parts of the map the player has not yet explored
  • requires to scout unexplored areas to find enemy sources

Micromanagement

  • way of controlling units in detail while they are in combat

Tactics

  • a set of specific actions used when applying a strategy

Strategy

  • making decisions knowing what we saw from the opponent

Example: Starcraft 2 Tech-tree

Starcraft II Tech-tree

RTS AI Layers

  • micromanagement
  • tactics (army positions)
  • strategy (tech tree)
RTS AI layers

Layer-based AI

  • each layer handles specific task
  • decision components consider available information, retrieved by lower components, to make decisions; executors are triggered afterwards
  • Sensor - reads the game state
  • Analyzer - combines sensing data to form a coherent picture
  • Memorizer - stores various types of data (terrain analysis, past decisions,...)
  • Decider - a strategic decider, determines goals
  • Executor - translates goals into actions
  • Coordinator - synchronizes groups of units
  • Actuator - executes actions by modifying the game state
Layer-based AI in games

Example: Layer-based AI

Hero
Layer Based AI diagram

Example: Layer-based AI

Layer Based AI diagram
Cossacks 3

Example: Megaglest

  • Open-source 3D RTS (LINK)
  • seven factions: tech, magic, egypt, indians, norsemen, persian, romans
  • rule-based AI
Megaglest

Megaglest architecture

Megaglest architecture

Megaglest rules

RuleConditionCommandFreq [ms]
AiRuleWorkerHarvestWorker stoppedOrder worker to harvest2 000
AiRuleRefreshHarvesterWorker reassigned20 000
AiRuleScoutPatrolBase is stableSend scout patrol10 000
AiRuleRepairBuilding DamagedRepair10 000
AiRuleReturnBaseStopped unitOrder return to base5 000
AiRuleMassiveAttackEnough soldiersOrder massive attack1 000
AiRuleAddTasksTasks emptyAdd tasks5 000
AiRuleBuildOneFarmNot enough farmsBuild farm10 000
AiRuleResourceProducerNot enough resourcesBuild resource producer5 000
AiRuleProducePerforming prod. task2 000
AiRuleBuildPerforming build task2 000
AiRuleUpgradePerforming upg. task30 000
AiRuleUnBlockBlocked unitsMove surrounding units3 000

Lecture Summary

  • I know what challenges in terms of game AI the developers face
  • I know basic AI techniques, such as scripting, FSM, and HFSM
  • I know something about behavior trees
  • I know the definition of supervised, unsupervised, and reinforcement learning
  • I know the main element and features of RTS