This morning I attended a talk given by Glen Van Datta, Director of Online Technology (SCE-RT) at Sony Computer Entertainment America (SCEA). His presentation was to the GamePipe Lab at USC and was about MMOG Technology Strategies that they use at SCEA.
Keep in mind that SCEA and SOE (Sony Online Entertainment) are pretty much separate companies and SOE does most of the MMORPGs (Everquest et al) but SCEA does do a lot of multiplayer online games for the Playstation systems including some upcoming larger MMOGs like the PlayStation Home virtual world. Thus the talk was not just about strategies for large persistent worlds but also for smaller world instances you may find in a typical multiplayer game.
Lots of this stuff is fairly technical and focuses on networking technologies and such but it’s certainly useful to know for anyone developing multiplayer games. This post will mainly be an overview of what was talked about and thus an overview of technologies for MMO and multiplayer games. I intend to go into much more detail about certain things aspects in later posts when I start wading more into this stuff myself.
Also one thing that was particularly interesting that was mentioned was that through experience the industry has learned that if one person cheats in an MMO game or is even perceived to have cheated to gain an unfair advantage, the game is likely to lose 100 players because of this one person.
MMOG Technology Strategies
The first thing to consider is the architecture of your typical modern MMOG. There is a game server that holds the world state that all of the clients will connect to, there is also a lobby server that the clients will connect to as well that does the matchmaking for the game. Sometimes there is also a community server which basically is a web based aspect to the game (such as the Armory for World of Warcraft) that players may connect to over the internet while they may not playing the game.
Typically the developer will want to implement a reliable UDP system as TCP systems will generally not work well for real time games because of the bandwidth used to ensure packets arrive and in the correct order. Reliable UDP allows the developer to decide which packets are important enough to depend on and which can be safely ignored if they don’t arrive.
A reliable UDP system may provide a categorization of packets that allows the server to label packets as in-order, reliable or latency critical among other useful categorizations.
The typical connection in the US runs at 256KB/second up and 750KB/second down.
The filtering system on the game server allows for networked LOD (level of detail) thus the server may choose not to send all information to a player. For example if a player can’t see another player they may not need to know all the information about that other player, or if a player can only see another player far in the distance then ever piece of information about what that player is doing may not be relevant. The filtering system allows the game server to make these determinations and save bandwidth by not sending irrelevant information to a player.
The filtering system can run at the sight level, the sound level or even the game object level where a particular game object may know that it doesn’t need very specific information about some other type of object. For example a Commander doesn’t need to know the specific details of each of his infantry units positions and movement trajectories but each of those infantry soldiers would since they are marching with one another.
Global Time Base
The global time base is a process by which all clients and the server are synchronized to some global time with some small acceptable variance. This allows for deterministic client calculations of when things should occur or end (for example you may receive the Blessing of Salvation buff in World of Warcraft, your client knows this lasts for 5 minutes and thus can show you when it ends without having to be notified by the server.) This can also be used to make position predictions based on trajectory of items in a game like bullets or other things as long as no other forces are acting on them that you don’t know about.
The global time base can also help when leveraging deterministic randomness, which is when a seed value is distributed to each client for their random number generator to make sure they are all synchronized. This ensures that for example the 3rd random number generated on each client that had the same seed value will be identical.
The global time base as well as many of these other technology strategies is also not needed for turn based or lock step gameplay.
Arbitration simply means that the server has to keep track of the state of the world and handle any inconsistencies in the clients views. For example one client may thing a door is open while another thinks a door is closed, without some way to figure this out this can be very important (depending on the game of course.) The server is responsible for pulling together all the information it can to solve these disputes, whether that means polling the clients to vote or keeping full track of the world state.
This stuff has to be considered up front and is a very important aspect for the game designers to consider. The developers must figure out what types of things are important and what are not, usually whether a player is dead or not is an important decision but whether or not some grass has been burned away or some leaves are blowing in the wind may not be as important to the users experience. Thus it is important to make these considerations during design so that the arbitration systems can be implemented as needed.
Distributed Game Servers
Game servers can become very loaded depending on how heavily trafficked your game has become. Some things can be either run together on the game server or offloaded onto external distributed servers to do the processing work needed. These things include NPC state information, AI behaviors, Scribe functions (logging, persistence information, debugging information), collision calculations and also dynamic effects such as polygon mesh changing (fully dynamic terrain meshes.) These are not the only things that can be offloaded, basically anything which you can write a subsystem for can usually be offloaded onto distributed systems that will scale better than trying to improve the capacity a single server box.
The lobby servers are the servers that perform matchmaking functions for the players as well as allow them to chat with one another and other functions. Lobby servers are typically less latency critical and are much more tolerant of lag and network traffic spikes, they also tend to use much less bandwidth. Usually TCP communication will be sufficient for lobby servers because of the somewhat non realtime nature of their functions. Lobby servers are also generally used to house friend and ignore lists as well as anti cheat mechanisms such as piracy detection and modded console detection.
The web interface to the game is classified as the communication server. These can be wide and varied and can include many different things. For instance World of Warcraft provides the Armory as a window into their game world from from the web. From the Armory you can view players equipment, character builds and also information about their guild and lots of other stuff. Forums for games also tend to fall into this category. These services provide a way for the players to be involved with the game while away from the game, or to arrange groups and things for when they are playing. Web interfaces to MMO games are becoming a standard thing and tend to provide a good value and sense of attachment to a game for your players. Most any new MMO game would feel incomplete without some web infrastructure.
Testing and Production
Lastly testing is an important aspect to any game development but is especially important in MMO games where many players will be using your systems simultaneously and where exploited bugs are much more detrimental to a larger group of players.
Also production times are another important thing to consider. When you plan to develop a large MMO game you have to realize that production can take a long time, 3 years or more. Because of this it is very important to spend some time at the beginning really laying out your plans and considering what is the riskiest part of your technology plan. This time period will typically be 6 months or so and during this time you should do things such as develop gameplay prototypes, work on the riskier aspects of the game to determine whether your feasibility estimates are realistic or not and ultimately decide if you should invest much more time and money into the project after this initial stage.
Ultimately MMOG development is a very complicated process. A multitude of technologies go into the creation of any multiplayer game and it is important for the involved in the development to have a good understanding of all of these technologies going into the project.
Well that’s about it, I think it was a good talk overall about the technologies going into MMOG games and I hope I’ve been able to pass some of the wisdom on to you. Let us know about any games you’re involved in making we’d love to hear about them.