FIT CTU

Adam Vesecký

vesecky.adam@gmail.com

Lecture 11

Multiplayer

APH.DODO.ME

Architecture of Computer Games

Adam Vesecký

Literature

Networking Architecture

Multiplayer

  • offers a capability for multiple players to act within the same world at the same time
  • complex feature that has a significant impact on the game architecture, touching almost every fundamental structure

Multiplayer categories

  • single-screen multiplayer
  • split-screen multiplayer
  • networked multiplayer
  • MMOG
  • Cloud Gaming

Multiplayer history

Local multiplayer games

  • Tennis for Two (1958)
  • Spacewar! (1962)

Networked multiplayer games

  • Empire (1973)
  • Maze War (1974)

Local area network games

  • Doom (1993)
  • Duke Nukem 3D (1996)

Online games

  • Quake (1996)
  • Unreal (1998)

MMO

  • Ultima Online (1997)
  • World of Warcraft (2004)

Libraries

Torque Network Library (opentnl)

ReplicaNet

  • powerful library for object sharing (sessions)
  • expensive

RakNet

NetStalker

Issues

  • the main objective - how to synchronize several universes
  • all clients have to achieve a certain degree of synchrony
  • attributes that affect the gameplay: capacity, speed, latency, jitter, packet loss
  • there is no known real-world picture for this type of problem
  • it's easy to convert multiplayer game into a single-player game
  • it's very difficult to implement multiplayer features into a single-player game

Methods

  • transfering a complete game state to all clients - for simple games
  • transfering a minimal subset for recostructing the complete information - more common

Topologies

  • peer-to-peer
  • client-server

Peer-to-peer architecture

  • each device exchanges data with each other in a fully connected graph
  • used in Doom, early Command & Conquer series, Age of Empires, Starcraft
  • given  peers, each must have  connections ->  in total
  • methods: single master, partial authority, full replication

Peer-to-peer architecture

Single master

  • one machine is chosen to be the game master

Partial authority

  • certain peers have authority over certain parts of the game
  • better bandwidth and processing power balance
  • difficult to implement and debug
  • each dynamic object belongs to exactly one machine
  • if one player drops out the game, all of the objects over which it had authority must be picked up by other machines

Full replication

  • each peer executes the game logic and only events are exchanged
  • hard to synchronize to avoid alternate realities
  • difficult to get the same results (floating point calculations)
  • runs well for a few players over LAN but terribly over the internet

Peer-to-peer games

Doom (1993)

  • 14.4 kbps PPP or 28.8 kbps SLIP
  • each turn player inputs were exchanged with other peers
  • you had to wait for the inputs from the most lagged player
  • every 30ms, the input from each player is sampled into a tic command
  • when the tic commands for all players have been received, the game advances its state

Age of Empires (1997)

  • for 8-player battle there could be up to 400 units
  • used Turn Timer - queue for commands
  • AoE synchronizes the commands each player issued, rather than units
  • all commands during 200ms are saved into a buffer
  • when the 200-ms-frame is over, all commands for a player's turn are transmitted to others

Client-Server architecture

  •  devices,  connections
  • server must handle  more messages per second
  • server quickly becomes the bottleneck (lack of power and bandwidth)
  • Dedicated server - only runs the game state and communicates
  • Listen server - server is an active participant in the game itself

Example: Quake

  • the first game that used partial reliability
  • each client acts as a dumb terminal
  • outputs are picked up by the prediction layer
  • the server runs at 20 FPS, while the client runs at 60 FPS

Command

  • the elementary unit of communication
  • used to udpate a position of the player, orientation, HP,...
  • reliable commands (with impact on the game state) have to be confirmed

NetChannel header

Example: Starsiege Tribes

  • sci-fi FPS, released in 1998
  • highly optimized networking model
  • supported up to 128 players over LAN or modem (56.6 kbps)

Data categories

  • non-guaranteed data: nonessential to the game
  • guaranteed data: guarantees both arrival and ordering (events)
  • guaranteed critical data: highest priority (player movement)
  • most recent state data: hit points

Example: Starsiege networking model

  • Game Simulation Layer - determines what particular client needs to know
  • Ghost Manager - replicates or "ghosts" dynamic objects that are relevant
  • Move Manager - transmits movement data of the player as quickly as possible
  • Event Manager - maintains a queue of events generated by the simulation
  • Stream Manager - sends data to the connection manager
  • Connection Manager - manages a notification of delivery status
  • Platform Packet Module - socket API wrapper, implements reliability layer

Transport

Transport layer

  • TCP - Transmission Control Protocol

    TPC header

  • UDP - User Datagram Protocol

    UDP header

  • DCCP - Datagram Congestion Control Protocol
  • RUDP - Reliable User Datagram Protocol
  • CUSP - Channel-based Unidirectional Stream Prot.

Ports

  • Well-known ports: 0-1023
    • 666: Doom
  • Registered ports: 1024-49151
    • 2302: Arma
    • 6112: Battle.net
    • 27015: Valve Servers
    • 27500: Quake
    • 28960: Call of Duty
  • Private ports: 49152-65535

TCP vs UDP

TCP

  • header size 20-40B
  • connection-based (requires 3 packets to set up a connection)
  • guaranteed reliability and proper ordering
  • may be prioritized over UDP packets
  • allocates a lot of resources to manage connections

UDP

  • header size 8B
  • lightweight and fast
  • no concept of connection
  • no guarantee of reliability of ordering
  • no flow control
  • may be dropped before TCP packets

TCP vs UDP

  • as opposed to SW applications, games have two main types of messages
  • changes to the gameplay (actions, events) need to be confirmed
  • streaming data (positions) don't need to be confirmed
  • critical events may have higher priority over others
  • UDP represents a basic streaming protocol onto which engines build a customized gaming protocol
Using TCP is the worst possible mistake you can make when developing a multiplayer game.Glenn Fielder, 2008

Multiplayer engine architecture

  • the server is running the game and clients only process inputs and display results

Issues

  • each player has only partially consistent view of the game world
  • in some cases, the server may disagree with the client's state
  • there is a delay between taking an action an observing the result

Transfer Control

Connection

  • when a client wants to join a game, it sends a hello packet to the server
  • once the server receives the hello packet, it assigns a player ID and sends a welcome packet to the client, containing the assigned ID
  • when the client receives the welcome packet, it saves its player ID and starts sending and receiving replication information to the server

Reliability

  • we have to identify packets by using sequence numbers and send them back to the server
  • example: ACK field in the message headers
  • issue: if the server sends 30 PPS and the client can only send 10 PPS, we need at least 3 ACKs included in each packet

Flow control

  • clients use basic congestion avoidance algorithms
  • servers can change streaming frequency based on the situation

Example: Message Header

  • SEQ number - sequential number of the message
  • ACK number - number of another message to be confirmed
  • ACK bit array - confirmation flags for previous 32 messages
  • Type - type of the message (update, disconnect, command,...)
  • ActionID - id of particular action

Message Types

Stream

  • doesn't need to be confirmed, contains a collection of continuous values
  • e.g. dynamic objects and their attributes (transformation)

Snapshot

  • complete information of the game state, sent either on demand or at given intervals

Command

  • messages that have an impact on the game state, have to be confirmed
  • e.g.: UNIT_CREATED, UNIT_DESTROYED, BUILDING_COMPLETED

Action

  • high-priority messages (player's inputs, fire button,...)

Procedure Call

  • a generic message that allows to call any function (play sound, load assets, reset animation)

Connection messages

  • messages for handshake, ID assignment, disconnect, etc.

Beacon

  • regular messages to inform the server that the connection is still on

Example: Goat Attacks

Example: Goat Attacks

Server-Client messages

  • GPCIdentifyPlayer
  • GPCReady
  • GPCAddPlayer
  • GPCRemovePlayer
  • GPCSpawnNPC
  • GPCRemoveNPC
  • GPCChatMessage

Client-server messages

  • GPSRespawnRequest
  • GPSJoinRequest
  • GPSChatMessage
  • GPSBomb
  • GPSPlayerChanged

Serialization

  • a large game world can have hundreds of moving objects
  • we need to remove any information that doesn't need to be sent
  • messages should be as close in size to MTU (~1500B) as possible

Example with no optimization

  • RTS battle
  • 5 players
  • 500 moving units
  • each unit has 20 attributes of 32-bit size -> 80 B per unit
  • the server sends 30 messages per second
  • header size is 42B (IP + UDP + networking header)
  • required bandwidth for the server:  Mbps

Serialization - Binary Footprint

  • we serialize everything - not a robust solution

struct Mage {

    int health;

    int mana;

}

 

void Serialize(const Mage* mage) {

    SendMessage(reinterpret_cast<const char*>(mage), sizeof(Mage));

}

 

void Deserialize(const Mage* output) {

    ReceiveMessage(reinterpret_cast<char*>(output), sizeof(Mage));

}

Health = 10, Mana = 14; Little Endian

Serialization - Streams

  • streams allow us to customize what attributes to serialize and how
  • better solution for collections

 struct Mage : public Streamable {

    vector<Item*> items;

    int health;

    int mana;  

  

    void SaveToStream(NetWriter* writer) {

        writer->WriteDWord(health);

        writer->WriteDWord(mana);

        writer->WriteDWord(items.size());

        for(auto item : items) item->SaveToStream(writer);

    }

  

    void LoadFromStream(NetReader* reader) {

        health = reader->ReadDWord();

        mana = reader->ReadDWord();

        int itemsCount = reader->ReadDWord();

        for(int i=0; i<itemsCount; i++) {

            items.push_back(new Item(reader));

        }

    }

}

Compression

Compression of bits

  • it is helpful to represent values with as few bits as possible
  • we can work with limited range and precision

Entropy Encoding

  • we compress data based on how unexpected it is
  • example: we can assume that the rotation is 99% of its time equal to zero, hence we can use only one bit to indicate this

Compression

Compression of attributes

  • we can serialize only attributes that vary
  • each object has a bit field that indicates which attributes follow in the stream

Compression of the payload

  • Huffman encoding, run-length encoding, LZ4,...

Compression

Delta messages

  • if a number doesn't change much, nor will its bits
  • between two frames, there is only a few changed bits for common attributes (position, rotation,...)
  • we store only differences by using XOR operation and compress the data
  • very effective but sensitive to packet loss
  • we can store 2/3 of attributes as diffs and 1/3 as full data - we will only need 3 consecutive packets to retrieve the full state

Replication

  • the act of transmitting a state of an object from one device to another
  • each object must be uniquely identified (network ID)
  • the network message contains a type of an object and all parameters required to construct it

 switch(actionType) {

    case OBJECT_CREATED:

      int objectType = reader->ReadDWord();

      auto factory = creatorManager->FindObjectFactory(objectType);

      auto newInstance = factory->CreateInstance(reader); // parse parameters

      objects.push_back(newInstance);

    break;

    case OBJECT_DELETED:

      int objectId = reader->ReadDWord();

      sceneManager->removeObjectById(objectId);

    ...

  }

Reliability

  • packets may get lost
  • server keeps sending messages that have an impact on the game state until the client accepts them

Ordering

  • packets may arrive in a different order
  • the client shouldn't apply a command message to its state before it applies all previous messages

Latency

Latency

  • the amount of time between an observable cause and its observable effect
  • e.g. mouse click and a unit responding to its orders

Suitable latencies

  • FPS: 16-60 ms
  • RTS: < 250 ms

Non-network latency

  • input sampling latency (~2 ms)
  • rendering pipeline latency (< 16 ms)
  • frame decoding latency for cloud gaming (~2-16 ms)

Network latency

  • processing delay (encryption, routing)
  • queuing delay (router can only process a limited number of packets at a time)
  • transmission delay (information can't travel faster than the speed of light)

Latency Example

  • both clients have the same latency

Latency Example

  • client B has a higher latency

Synchronization

Interpolation

  • client can run at 60 FPS, the server usually sends state updates at 10-30 FPS
  • Interpolation: whenever the client receives a new state, it smoothly interpolates to that state
  • Problem: the object might change its interpolated value instantly (teleport)

Interpolation Reserve

  • client renders a state that is 2 frames old
  • smooth interpolation, but we create artificial delay

Extrapolation

  • without interpolation reserve, the client tries to extrapolate
  • results in jittery animation

Deterministic prediction

  • interpolation reserve is precise but delayed behind the server
  • the client is always at least 1/2 RTT behind the true state
  • some attributes can be handled on client side completely (e.g. camera pose in FPS)

Prediction

  • the client predicts the future value by running the same simulation code
  • to perform extrapolation by 1/2 RTT, the client must approximate the RTT

Non-deterministic prediction

  • non-deterministic values are hard to predict (steering behaviors, other players' movement,...)
  • solution: make an educated guess and correct it when an update arrives
  • Dead reckoning - process of predicting a behavior of an entity based on the assumption that it will keep doing what it's currently doing

Server simulation

Client misprediction

Dealing with correction

  • Instant state update - updates immediately when the correct value arrives
  • Interpolation - smoothly interpolates to the correct value
  • Second-order state adjustment - adjust acceleration instead of velocity

Server-side rewind

  • dealing with instant actions that affect the gameplay (e.g. instant hit in FPS)
  • occurs due to the inaccuracies of dead reckoning and time dilation
  • server may change a state that has already been confirmed

Source engine's solution

  • rewinds state on the server to exactly the state in which the player fired
  • server stores the poses of every relevant object for X last frames and looks up the two frames between which the client was interpolating

Wrong

Correct

Example: Server-side rewind

  • Client B has 3x higher latency than Client A
  • Client B performed a change before Client A did

Latency handling summary

Time dilation

  • delays the values by a few frames and interpolates to them

Deterministic prediction

  • runs simulated code, masks latency and keeps the client's state in sync

Dead reckoning

  • non-deterministic prediction
  • client uses the last known state of an object to extrapolate future state

Server-side rewind

  • the server buffers object positions for several frames to match the client's view when processing instant events
It is better to be wrong on time than right but late

Example: Source engine

  • server simulates the world at 30-60 FPS and sends 20 snapshots per second by default, using an interpolation reserve of 100 ms
  • client samples inputs at 30-60 FPS and buffers snapshots for 100 ms
  • server uses time dilation and rewind

MMOG

MMOG

  • Massively Multiplayer Online Game
  • Ultime Online, World of Warcraft, Guild Wars, Skyforge, Elder Scrolls Online
  • interaction is not really massive
    • players can only interact with a limited amount of players at the same time
    • when more than expected number of players gather together, the game may crash

Swifty Invasion in WoW

Object Relevancy

  • an object is considered relevant for a client when it should be informed about its updates
  • in a MMOG, it may not be important to know about players several miles away
  • similar sort of problems as visibility culling (or spheres or influence)

Static zones

  • splits the world into regions and assigns a separate process/server to each region
  • only objects in the same zone as the player are relevant

Server partitioning

  • splits the world into completely isolated regions
  • e.g. islands, continents

Instancing

  • one shared game supports several separated instances at once
  • example: dungeons in WoW, Dota/LoL tournaments

Example: Ultima Online

  • network is based on Pastry DHT (Distributed Hash Tables)
  • world is split into regions of size based on the amount of players
  • each region is assigned an ID and mapped into the DHT
  • each node is responsible for exactly one region

3 data types

  • static objects - immutable part of the world (terrain)
  • dynamic objects - interactive items and NPCs, deterministic
  • avatars - players, nondeterministic

Security

Classic threats

  • packet sniffing, man-in-the-middle
  • ghosting - scouting the players using multiple connections

Input validation

  • players can't perform an action that is invalid
  • only a client responsible for Player A can send an action that affects that player
  • client should validate the messages from server as well

Software cheat detection

  • actively monitors the integrity of the game
  • cheating SW can hook into the game, overwrite memory, modify files
  • map hacking - removing fog of war and revealing resources
  • bot cheat - bot that either plays the game or assists the player
    • e.g. dummy levelling, aimbots
  • Valve Anti-Cheat - available for games that utilize Steamworks SDK
    • maintains a list of banned users, scans for known cheat programs
  • Easy AntiCheat - prevents cheating on a technical level

Lecture Summary

  • I know the difference between P2P and client-server architecture
  • I know what issues networking architecture may face
  • I know what types of messages can be used in multiplayer games
  • I know what replication is
  • I know how reliability in multiplayer works
  • I know how interpolation works
  • I know what server-side rewind is
  • I know how games can deal with latency

Goodbye Quote

A hero needs not speak. When he is gone, the world will speak for him.Halo 3