Load Balancing

Fortunately, the arcane subject of load balancing is not something with which designers need concern themselves directly. However, they do need to be aware of the consequences of whatever solution is adopted by the technical experts, especially because they are likely to be consulted on the matter.

Ideally, a virtual world would run on a single, very powerful computer. For textual worlds, this is already the case ¹. Graphical worlds may go the same way ², but for the moment there are still plenty of things on which newly available computational resources could be spent at the server side:

• Increasing the number of players present in each incarnation (100,000 players on 50 servers is one thing, but 100,000 players on 1 server is something else).

• Increasing the level of detail at which the virtual world functions (for example, leaving footprints in the snow that fade as it melts or as new snow falls).

• Improving the AI of the virtual world’s denizens (both in quality and quantity).

It is therefore likely that virtual worlds will soak up whatever additional computing power is thrown their way for quite some time. Inexpensive machines will still be clustered, because it will always cost less to use eight computers of power X than to use one computer of power 8* X (although management overheads degrade overall efficiency as new machines are added, which is why no one would use 64 computers of power X/8) ³.

So, given that distributed servers are here to stay, how does this affect the designer?

Well, the virtual world is too big to fit on one computer; therefore it must be partitioned over several computers.

Sort of…

If the virtual world were inert, that is, nothing ever happened in it, it wouldn’t need any computers at all, it would just sit on a disk pack. It only needs computers when things happen in it. The issue is therefore one of ensuring that activity is spread across sub-servers such that they can all cope with the work they have; in other words, the computational load must be shared in a balanced way (hence load balancing).

The greatest source of activity in a virtual world is the player community. Every moment that they are in the virtual world, players are interacting with it. Merely moving from one location to another entails informing every other character that can (in the virtual world) see you do so. Yes, some activities are more of a drain on resources than others, but they tend not to be specific to particular playing styles. The issue is the sheer quantity of actions being performed, not the efficiency of individual actions. Load balancing in virtual worlds therefore generally means ensuring that roughly the same number of players is connected to each sub-server in the cluster.

The obvious way to do this would be to assign each incoming player to whichever sub-server has the fewest players. It turns out that it’s quite difficult to do this without introducing big overheads, though. To update the instantiation database to reflect an action, records need to be locked to prevent other sub-servers from also changing them at the same time (for example, if two characters attempted to pick up the same object simultaneously). The sub-server needs to lock all records that it could need during an action, perform the tests to ensure that the action is possible, make any necessary update requests, and unlock the records. This is a lot of locking/unlocking. It would be really handy if there were some way to block-book records in the instantiation database for long periods without relinquishing control of them. Are there any types of records for which this could easily be done?

Statistically, most actions performed by players involve movement ⁴.

The player wants to move their character from A to B, so the sub-server has to lock location B, check if it’s empty, if so then move the player into location B, then unlock it. Location A must also be locked, so anyone wanting to do anything to the character that assumes it is in location A (for example, teleport it to location C) will not inadvertently screw up things. Many other common commands (particularly get/drop and those to do with communication) are also location-based.

For these reasons, servers typically partition responsibility for the virtual world along (virtual) geographic lines. A sub-server can lock location records in the instantiation database for extended periods; indeed, if the system is programmed correctly, it doesn’t have to lock them at all—it has implicit control by mere virtue of the fact that none of the other sub-servers do.

To summarize the argument so far: We want to spread the players fairly evenly across sub-servers, but the obvious way would introduce too many overheads on database access; a far more efficient way to do it is to partition by geographical location. The question is: Would using this partitioning model give us load balancing?

The answer is that yes, it would. How, exactly, depends on the virtual world.

There are essentially two approaches: fixed load balancing and dynamic load balancing. The former, exemplified by EverQuest, assigns a predefined geographical location (a zone) to one sub-server ⁵; the latter, exemplified by Asheron’s Call, moves responsibility for geographical locations between sub-servers.

Fixed load balancing:

• Is easier to implement.

• Can partition the instantiation database and keep it local to the sub-server, for greater efficiency.

• Allows the client to figure out in advance what texture maps will be needed and preload them into graphics card memory.

Dynamic load balancing:

• Has seamless terrain (you can see to the horizon ⁶).

• Has boundaries that are not physical (monsters chasing you don’t get stuck at zone edges).

• Balances the load better.

What does all this mean for design?

It’s an example of where technology imposes constraints. For fixed load balancing, zones can be created with greater individuality: The ‘physical’ barriers between them allow for radical change. Players can cross from one to another and expect to see something different on the other side. For a seamless system, sudden changes have to make more sense or they’ll seem out of place.

The fact that content such as monsters can’t cross zone boundaries means that players will use different tactics in such a world than they would in one where there was always the possibility that a creature they had royally annoyed could pursue them relentlessly.

Victorian London’s police forces couldn’t (legally) cross precinct boundaries: Jack the Ripper would commit a murder in one police district and then run into another where the police were not allowed to follow; if they could have followed, he would have had to rethink his getaway strategy.

Zoned worlds have something of a problem with fiash crowds (people appearing instantly in the same vicinity in response to interesting news; the term comes from a 1973 Larry Niven short story ⁷). Most of the time, each sub-server will be handling similar numbers of players. However, sometimes something happens that causes everyone to want to be in the same general locale ⁸. Maybe it’s a rare spawning of an impressive dragon, or a social event such as a wedding or guild rally. Whatever, all of a sudden more people want access to a server than it can handle. The designer has to decide what to do when this happens. Do they simply show a “zone full” message if people try to enter it? Do they let them in and leave Customer Service to handle the resulting complaints about lag? Do they organize the virtual world such that it would be counter to its fiction for everyone to want to be in the same zone?

Dark Age of Camelot partitions its player base into three “realms.” Members of one realm can’t enter territory belonging to members of another realm. They can enter a no-man’s land between two realms, however, which is where realm versus realm combat takes place. The upshot of this is that unless there is a serious skewing of the DAoC player base, only a third of the players online will usually be present in any one realm. That’s excellent for load balancing. What’s more, realms are aggregations of zones, they’re not zones in themselves. Each realm is made up of 13 outdoor zones (64K by 64K squares) plus five dungeons plus one city. There’s no reason why a sub-server has to handle zones from only one realm; if (for some reason) 75% of the players are all in one realm, the sub-servers handling that realm will automatically have less load from the other (sparser) realms they’re controlling.

Players of EQ notice how the sub-servers take responsibility for zones, but players of DAoC don’t. Why not? Because of the latter game’s world design.