Designing a Distributed System for an Online Multiplayer Game — Game Server (Part 5–1)
This is part number five of the Garage series, you can access all parts at the end of this post.
Before following up on the pipeline, we start with the game server architecture first.
Networking
There are two channels to communicate with the game server, the reliable channel to publish the game events (which need to be delivered to the game client with a guarantee and in order), and a fast channel to stream the game world states (which are called snapshots).
TCP Socket
For the main events of the games, we use the TCP protocol to guarantee that all events are delivered without corruption and in order. The WebSocket is useful here, it upgrades an HTTP request as a long-living connection and uses it to read and write bytes.
UDP Socket
All snapshots and users’ inputs must be delivered as fast as possible without any acknowledgment or latency overheads. The UDP is not aware of any connection and is stateless, but we need a virtual connection here to register and authenticate the user and send messages to it, to achieve this, I developed an open-sourced package to manage the UDP clients. It initiates a virtual connection between the server and the client. We’ll check this out in detail after other challenges.
Security
There is a common challenge for both TCP and UDP channels, security. All packets are transferred over the internet and they can be monitored by the man-in-the-middle. Thus, every data must be encrypted to be safe.
Note that the game client connects to the game server directly via IP. Kubernetes runs the dedicated game server as a pod that uses node network namespace to access the internet. Eventually, we have the node public IP and the dedicated game server listening ports. We don’t want any overhead latency. for TLS we need a domain for each node and we can’t use the domain because of DNS resolving overhead (assigning a domain name to scalable nodes and issuing a TLS certificate for each of them increases the complexity). TLS is used for the TCP channel and DTLS must be used for the UDP.
udpsocket
I developed an open-sourced UDP socket package to manage the clients & create a virtual connection:
Features:
- Handle handshake
- Authentication
- Manage clients & connections
- Encryption
- Implementable with any other encryption methods, transporting data structure protocols, and authentication policies
In a nutshell, this package helps to register and authenticate the clients and establish a virtual stateful secure connection with them. You can implement your arbitrary encryption methods, using any transporting data structure protocol (the protobuf is supported by default), or your authentication policies.
I tried to implement a mimic of DTLS in the udpsocket package for the UDP channel.
Encryption
Each game server has a unique random RSA private key. The client downloads the game public key using the game manager HTTP server over TLS, so the public key is downloaded secure and the client (player) must have access to the game to download it, however, it’s the public key and doesn’t matter. The game client uses this public key to encrypt its AES encryption key for the game server.
The encryption is applied to the WebSocket connection as well. The client encrypts its AES encryption key with the server RSA public key, in the following, it encrypts the user’s JWT token with the AES encryption key and passes them in the HTTP request cookies. The server decrypts the AES key with its private key, then decrypts the JWT token with the AES key, and finally authorizes and registers the client.
Authentication
I used Firebase for user authentication. After logging in to the game, it issues a token for each session, then the token is used to authorize user requests.
Encoding
To transport the data between the client and the server, we need to encode data before encryption. I used two different formats for UDP and TCP channels. as I mentioned before, the udpsocket package (which is used for the UDP channel) supports a custom implementation of the encoder, therefore I used the MessagePack as an encoder for the UDP and JSON for the TCP channel.
Event Broker
The game server publishes internal events to communicate with the game manager via the broker. Some events like GameServerReady with selected port numbers.
Pipeline
Let’s put these parts together and review the whole pipeline:
The game server tries to find a free port on the node to expose the TCP and UDP sockets. when the ports are bound, the game server publishes a readiness event with listening ports to the broker and waiting for the clients to connect.
The game manager which has been subscribed to the game events topic on the broker, receives the game server readiness event and caches the exposed ports, then it publishes an event to the game client with the game server public IP and ports.
On another side, the game client tries to download the game server RSA public key over TLS from the game manager. Here, the game client encrypts its secret AES key using the server public key and tries to establish two connections over UDP (using udpsocket protocols) and TCP (using WebSocket). In the following, the game server authorizes the requests and registers the client. After the client registration on the UDP connection, a temporary session ID is generated for further requests.
Now the client and the server connections have been established and they are ready to exchange messages, In the next part we’ll review the game logic & mechanics.
All posts:
