ZenAndGames.com

This morning I attended a talk given by Glen Van Datta, Director of Online Technology (SCE-RT) at Sony Computer Entertainment America (SCEA). His presentation was to the GamePipe Lab at USC and was about MMOG Technology Strategies that they use at SCEA.

Keep in mind that SCEA and SOE (Sony Online Entertainment) are pretty much separate companies and SOE does most of the MMORPGs (Everquest et al) but SCEA does do a lot of multiplayer online games for the Playstation systems including some upcoming larger MMOGs like the PlayStation Home virtual world. Thus the talk was not just about strategies for large persistent worlds but also for smaller world instances you may find in a typical multiplayer game.

Lots of this stuff is fairly technical and focuses on networking technologies and such but it’s certainly useful to know for anyone developing multiplayer games. This post will mainly be an overview of what was talked about and thus an overview of technologies for MMO and multiplayer games. I intend to go into much more detail about certain things aspects in later posts when I start wading more into this stuff myself.

Also one thing that was particularly interesting that was mentioned was that through experience the industry has learned that if one person cheats in an MMO game or is even perceived to have cheated to gain an unfair advantage, the game is likely to lose 100 players because of this one person.

MMOG Technology Strategies

The first thing to consider is the architecture of your typical modern MMOG. There is a game server that holds the world state that all of the clients will connect to, there is also a lobby server that the clients will connect to as well that does the matchmaking for the game. Sometimes there is also a community server which basically is a web based aspect to the game (such as the Armory for World of Warcraft) that players may connect to over the internet while they may not playing the game.

Game Server

Reliable UDP

Typically the developer will want to implement a reliable UDP system as TCP systems will generally not work well for real time games because of the bandwidth used to ensure packets arrive and in the correct order. Reliable UDP allows the developer to decide which packets are important enough to depend on and which can be safely ignored if they don’t arrive.

A reliable UDP system may provide a categorization of packets that allows the server to label packets as in-order, reliable or latency critical among other useful categorizations.

The typical connection in the US runs at 256KB/second up and 750KB/second down.

Filtering

The filtering system on the game server allows for networked LOD (level of detail) thus the server may choose not to send all information to a player. For example if a player can’t see another player they may not need to know all the information about that other player, or if a player can only see another player far in the distance then ever piece of information about what that player is doing may not be relevant. The filtering system allows the game server to make these determinations and save bandwidth by not sending irrelevant information to a player.

The filtering system can run at the sight level, the sound level or even the game object level where a particular game object may know that it doesn’t need very specific information about some other type of object. For example a Commander doesn’t need to know the specific details of each of his infantry units positions and movement trajectories but each of those infantry soldiers would since they are marching with one another.

Global Time Base

The global time base is a process by which all clients and the server are synchronized to some global time with some small acceptable variance. This allows for deterministic client calculations of when things should occur or end (for example you may receive the Blessing of Salvation buff in World of Warcraft, your client knows this lasts for 5 minutes and thus can show you when it ends without having to be notified by the server.) This can also be used to make position predictions based on trajectory of items in a game like bullets or other things as long as no other forces are acting on them that you don’t know about.

The global time base can also help when leveraging deterministic randomness, which is when a seed value is distributed to each client for their random number generator to make sure they are all synchronized. This ensures that for example the 3rd random number generated on each client that had the same seed value will be identical.

The global time base as well as many of these other technology strategies is also not needed for turn based or lock step gameplay.

Arbitration

Arbitration simply means that the server has to keep track of the state of the world and handle any inconsistencies in the clients views. For example one client may thing a door is open while another thinks a door is closed, without some way to figure this out this can be very important (depending on the game of course.) The server is responsible for pulling together all the information it can to solve these disputes, whether that means polling the clients to vote or keeping full track of the world state.

This stuff has to be considered up front and is a very important aspect for the game designers to consider. The developers must figure out what types of things are important and what are not, usually whether a player is dead or not is an important decision but whether or not some grass has been burned away or some leaves are blowing in the wind may not be as important to the users experience. Thus it is important to make these considerations during design so that the arbitration systems can be implemented as needed.

Distributed Game Servers

Game servers can become very loaded depending on how heavily trafficked your game has become. Some things can be either run together on the game server or offloaded onto external distributed servers to do the processing work needed. These things include NPC state information, AI behaviors, Scribe functions (logging, persistence information, debugging information), collision calculations and also dynamic effects such as polygon mesh changing (fully dynamic terrain meshes.) These are not the only things that can be offloaded, basically anything which you can write a subsystem for can usually be offloaded onto distributed systems that will scale better than trying to improve the capacity a single server box.

Lobby Servers

The lobby servers are the servers that perform matchmaking functions for the players as well as allow them to chat with one another and other functions. Lobby servers are typically less latency critical and are much more tolerant of lag and network traffic spikes, they also tend to use much less bandwidth. Usually TCP communication will be sufficient for lobby servers because of the somewhat non realtime nature of their functions. Lobby servers are also generally used to house friend and ignore lists as well as anti cheat mechanisms such as piracy detection and modded console detection.

Communication Servers

The web interface to the game is classified as the communication server. These can be wide and varied and can include many different things. For instance World of Warcraft provides the Armory as a window into their game world from from the web. From the Armory you can view players equipment, character builds and also information about their guild and lots of other stuff. Forums for games also tend to fall into this category. These services provide a way for the players to be involved with the game while away from the game, or to arrange groups and things for when they are playing. Web interfaces to MMO games are becoming a standard thing and tend to provide a good value and sense of attachment to a game for your players. Most any new MMO game would feel incomplete without some web infrastructure.

Testing and Production

Lastly testing is an important aspect to any game development but is especially important in MMO games where many players will be using your systems simultaneously and where exploited bugs are much more detrimental to a larger group of players.

Also production times are another important thing to consider. When you plan to develop a large MMO game you have to realize that production can take a long time, 3 years or more. Because of this it is very important to spend some time at the beginning really laying out your plans and considering what is the riskiest part of your technology plan. This time period will typically be 6 months or so and during this time you should do things such as develop gameplay prototypes, work on the riskier aspects of the game to determine whether your feasibility estimates are realistic or not and ultimately decide if you should invest much more time and money into the project after this initial stage.

MMOG Development

Ultimately MMOG development is a very complicated process. A multitude of technologies go into the creation of any multiplayer game and it is important for the involved in the development to have a good understanding of all of these technologies going into the project.

Well that’s about it, I think it was a good talk overall about the technologies going into MMOG games and I hope I’ve been able to pass some of the wisdom on to you. Let us know about any games you’re involved in making we’d love to hear about them.

Mike McShaffry a game industry vet has started a new blog to discuss things he’s learned through the years in the industry. Not much there to see yet but he does have one particular article on pitching your game that is very good. It details all the steps needed in a successful pitch and what questions and materials you need to be prepared with. It’s mainly focused on AAA games looking for 7 figure budgets but any indie who ever has to do a pitch for various reasons, whether getting some funding or finding a publisher, would do well to read it.

Matt Sakey has written a great article for the IGDA on the use of Lean methodologies for game development.

“Waste” in development could be rework, unused assets, reversed design decisions, creative clashes, you name it. Largely it would be the product of poor planning, which, I’m sorry to say, does lead right back to the “business needs more proven process models” argument.

Lean methodologies are used in production industries to reduce waste in the process and ultimately make better products for cheaper. I think this is a vitally important concept for indie game developers operating on small budgets and this article is certainly worth a read.

Eye of Judgment

I’ve been really stoked about Eye of Judgment since I heard about it and I still am now that it’s out. For those of you who haven’t heard of it, Eye of Judgment is a collectible card game (CCG) for the Playstation 3 (or is it PLAYSTATION 3) that allows you to view 3D renderings of your cards monsters and units and battle against the computer or other players online. It uses the Playstation Eye peripheral to view what cards you’re playing on the board and register them into the game world that is a virtual representation of the board. They also sell booster decks of the cards that you can purchase in game shops and such to make different decks. Wow that sounds complicated when I write it out. Perhaps this cheesy video will help.

Sadly, I don’t yet have a PS3 myself but I’m sure I’ll acquire one soon and this game is certainly putting the pressure on me. First you must understand that I’m a huge CCG fan and was once (and probably will be again in the future) a rather hardcore Magic The Gathering player, so I think about CCG games a lot and they’ve intrigued me for a long time. I’ve even got a game design I did for an online one sitting around that I hope to make one day.

Anyway this game sounds pretty great but I have heard negative aspects of how the game design isn’t that deep and the decks that come with the game are pretty bland so perhaps it’s not quite living up to it’s potential but it is certainly innovative and I’ll applaud that.

The Eye of Judgment Card Hack

The game only recently came out and some ingenious players have already figured out that they can hack it to be able to get any card in the game. Basically they scan the cards into their computer and print them out. This shouldn’t work and it’s an obvious method that the developers would have thought of before hand, and they did. You see they made some special “magical?” ink that they used to print the cards that the camera was supposed to detect, apparently though this doesn’t work at all and a normal inkjet printer can reproduce the cards well enough for the camera to recognize them and you can basically use any card and as many as you want using this technique. There’s more info on the printed cards here. It also looks like you may not even have to go through all that much trouble.

Obviously many people will think this is cheating unfair and such but I’m going to play devil’s advocate for a minute and explain why it’s not really that big of a deal. First of all any CCG designer worth his salt knows that your game has to be balanced under the assumption that all players will have unlimited access to your card base and therefore can (and will if it grants an advantage) play with the maximum number allowed of in their deck. The reason for this is because any hardcore fans of your game will go to any length to obtain exactly the cards they want for their deck and typically how difficult these are to obtain is irrelevant. This goes back to Magic and it’s limit of 4 of any one card in a deck, as I understand Eye of Judgment has a similar requirement, thus having all players have access to all the cards shouldn’t degrade from the entertainment and fun value of the game and will probably only put a hurt on the profit margins. Any big tournament level player of Magic typically doesn’t think about whether they have particular cards when considering what deck they want to play in an upcoming tournament.

Secondly, CCGs have a long tradition of proxying cards, proxying is the process which a player will take any mundane card write on it or otherwise mark it to indicate it actually represents some other card which they don’t actually own. This process originally necessitated because of the extremely rare power 9 cards and other rare cards in Magic the Gathering which can cost $1000+ each, obviously limiting them their accessibility to the greater base of players of the game and thus granting a somewhat unfair advantage to owners of these cards. Proxying Magic cards is a pretty standard practice these days and many tournaments even allow this practice for the older formats where the cards are much rarer, more expensive and hard for players to obtain because of the limited print runs of cards before the games popularity took off. Wizards of the Coast doesn’t allow proxying cards in any sanctioned tournaments however.

Now if you are considering playing Eye of Judgment online you must realize you will now be playing against players who have access to every possible card in the game in multiples. People will abuse this hack to gain an advantage because it is the nature of a player to take and use any advantage available to them, as Raph Koster tells us in Theory of Fun for Game Design. Now every player may not abuse this initially but some will, and since some will others who normally wouldn’t will be forced to abuse this hack to even the playing field. Eventually nearly every player will be abusing this and any new player to the game playing online must do likewise to keep up. This is a process that has occurred time and time again in multiplayer games and really society in general throughout history.

Ultimately the rules of the game change. No longer is it a game about collecting and skill begins to play a much larger role in the game in the long run since personal wealth and ability to acquire cards becomes a non-factor. What Magic has taught us though is that this isn’t really a bad thing and much fun can still be had when the game becomes a game of skill and less a game of chance, this is of course in theory as Eye of Judgment probably doesn’t have the depth and finesse that Magic has and ultimately the game design of Eye of Judgment and it’s ability to be a fun game will be the ultimate test.

It’s also possible that if they’re serious about this game they’ll figure out how to fix this with the next set or possibly even with some software update and the whole argument becomes a moot point until some fancier hacks are discovered. Either way this is an interesting story to watch.

If you are interested in CCG design then stick around I’ll certainly be discussing it more in the future. CCGs are one of my pet genres so I intend to come back to them quite a bit in future posts.

The IGF has revealed the entries for 2008 and there’s some sweet games on the list. I recommend checking it out.

I’m happy to see that Sector 13 is on the list, I briefly did some work with these guys on this game a while back but had to stop when I switched jobs. I know it’s come a long way since then and they’ve done a great job with it, I’d recommend checking them out.

I’m super stoked about Fallout 3 and love game design so this is right up my alley and possibly yours. The Fallout guys have decided to hold a contest where you design a perk for the game and the best one will be chosen and put into the game. I’ll probably try to come up with a few myself as this is pretty exciting. You can find the contest page here and I’ve pasted more info below.

Celebrating 10 Years of Fallout!

10 years ago the original Fallout was released and forever changed role-playing games. Now is your chance to be a part of Fallout history. We’re giving you, the fans, a chance to design one of the perks for Fallout 3!

No, we aren’t going to tell you about any of the perks that are already in Fallout 3. First, that would spoil it, and second, we don’t want to color your ideas with anything we are, or aren’t doing. It’s a clean slate for you. Be creative.

Think of something in the spirit of Fallout. What would you really want your character to be able to do in the game? Don’t worry about any stat requirements or the level you would get it at. Just design a cool perk and we’ll fit it in where it could go.

Part of what make Perks great are the Vault Boy images that goes with them. You can choose to use one of the generic icons we’ve provided to go along with your Perk if you’re not artistically inclined (that won’t hurt your chances of winning, in case you’re wondering) or you can choose to upload a drawing, sketch, photo, or creation of your own design. Send in a photo of yourself in the appropriate pose, sketch it on a napkin and scan it in…whatever you want.

If you win, you’ll get your Perk in the game, your name in the game’s credits, AND your choice of the grand prize from the Prize Vault: either the PC or the Console flavor.

All entries must be in by 11:59 pm (Eastern time) on October 31, 2007. We’ll announce the winner of the contest, and all the other awardees, in November.

PC Grand Prize:
Your choice of an ATI or NVIDIA video card
Logitech G15 keyboard
Logitech G9 mouse
Logitech G51 Gaming Speakers
Vault Boy bobblehead
Vault Tec lunchbox
Fallout 3 t-shirt
Vault Boy decal
OR
Console Grand Prize:
Xbox 360 Pro
Logitech G51 Gaming Speakers
Vault Boy bobblehead
Vault Tec lunchbox
Fallout 3 t-shirt
Vault Boy decal

We’ll have plenty of prizes for runners-up and honorable mentions, and we’ll also pick from all the entries at random to give away some additional prizes just so everyone has a chance to win something. Prizes will include great stuff from our friends at NVIDIA, ATI, Logitech, and Microsoft, as well as Fallout schwag.

You can enter as often as you like, but you can only enter the same perk once, and you can only win once. All entries subject to the terms and conditions of the contest. Make sure you read them before you enter.

Good luck to everyone. We look forward to seeing your ideas.

Happy 10th Birthday to Fallout.

Business Week has a great piece look at some basic fundamentals of game usability. They list 10 lessons which game designers need to know and should follow in their games. You can read the article here.

I think this is a fairly good list and is definitely something to keep in mind when designing your games. I can remember lots of times even I have just quit playing a game because of various usability problems where I otherwise probably would have continued playing.

The guys from Penny Arcade aren’t around this week and are having guests write their articles for them. Tycho from Penny Arcade ran into Raph Koster and posed a design challenge to him to come up with a MMO based around players building buildings, actually getting in there doing the wiring, plumbing the whole lot.

One thing Raph said that I really think is an important way of looking at games is this.

Games are made out of smaller games – turtles all the way down, until you hit the game that is so trivial and stupid it isn’t deserving of the name. In an MMO, we nest games pretty deep, because some games are short-term and some are long-term. In something like WoW, the smallest games are things like “hit Heal on time.” Then you get ‘kill the foozle,” and above that “kill a hell of a lot of foozles” and above that “make yourself stronger by picking the right gear” and in some cases “make your guild stronger” and so on.

I think that is a very insightful way of looking at how to design games, as mini games within themselves. The whole article is worth a read because the MMO he comes up with does sound like it could be kinda fun even. He came up with it in only 3 hours too.

I think the idea of doing mini designs and then showing them off is kind of cool, maybe I’ll do those occasionally here. Through up rough sketches of game ideas for anyone to read and pick apart that may be fun

Also Raph is working on a new project called Metaplace which looks to be very interesting. It promises to enable the players to design and create their own MMOs.

I just ran across this great resource for 3D models that would work great in indie games. The site is called 3DRT and they have a decent selection. They seam to have quite a few packs of models and many of the models come with several textures that work in making each one look really unique. Their prices are low as well and are thus fairly friendly to the indie developer. I haven’t used them myself but their stuff looks good so if you’re on the market for some 3D model packs these guys are worth checking out they may just have what you need.

Here’s a sample of one of their models with textures.