Inside Blizzard: Battle.net Skywing skywinguninformed@valhallalegends.com Last modified: 8/31/2005 1) Foreword Abstract: This paper intends to describe a variety of the problems Blizzard Entertainment has encountered from a practical standpoint through their implementation of the large-scale online game matchmaking and chat service, Battle.net. The paper provides some background historical information into the design and purpose of Battle.net and continues on to discuss a variety of flaws that have been observed in the implementation of the system. Readers should come away with a better understanding of problems that can be easily introduced in designing a matchmaking/chat system to operate on such a large scale in addition to some of the serious security-related consequences of not performing proper parameter validation of untrusted clients. 2) Introduction First, a bit of historical and background information, leading up to the present day. Battle.net is an online matchmaking service that allows players to set up online games with other players. It is quite possibly the oldest and largest system of it's kind currently in existence (launched in 1997). The basic services provided by Battle.net are game matchmaking and chat. The matchmaking system allows one to create and join games with little or no prior configuration required (other than picking game parameters, such as a map to play on, or so-forth). The chat system is similar to a stripped-down version of Internet Relay Chat. The primary differences between IRC and Battle.net (for the purposes of the chat system) are that Battle.net only allows a user to be present in one chat channel at once, and many of the channel parameters that IRC users might be familiar with (maximum number of users in the channel, who has channel operator privileges) are fixed to well-defined values by the server. Battle.net supports a wide variety of Blizzard games, including Diablo, Starcraft, Warcraft II: Battle.net Edition, Diablo II, and Warcraft III. In addition, there are shareware versions of Diablo and Starcraft that are supported on Battle.net, as well as optional expansions for Diablo II, Starcraft, and Warcraft III. All of these games share a common binary communication protocol that has evolved over the past 8 years, although different games have differing capabilities with respect to the protocol. In some cases, this is due to differing requirements for the game clients, but usually this is simply due to the older programs not being updated as frequently as newer versions. In short, there are a number of different dialects of the Battle.net binary protocol that are used by the various supported products, all at the same time. In addition to supporting an undocumented binary protocol, Battle.net has for some time now supported a text-based protocol (the ``Chat Gateway'', as officialy documented). This protocol supports a limited subset of the features available to clients using the full game protocol. In particular, it lacks support for capabilities such as account creation and management. Both of these protocols are now fairly well understood and documented certain persons outside of Blizzard. Although the text-based protocol is documented and fairly stable, the limitations inherent in it make it undesirable for many uses. Furthermore, in order to help stem the flood of spam on Battle.net, Blizzard changed their server software to prevent clients using the text-based protocol from entering all but a few pre-defined chat channels. As a result of this, many developers have reverse engineered (or more commonly, used the work of those who came before them) the Battle.net binary protocol and written their own "emulator" clients for various purposes (typically as a better alternative to the limited chat facilities provided by Blizzard's game clients). These clients emulate the behavior of a particular Blizzard game program in order to trick Battle.net into providing the services typically only offered to the game clients, hence the name ``emulator client''. Most of these clients area referred to as ``emulator bots'' or ``emubots'' by their developers, and the Battle.net community in general. In fact, there are also partially compliant server implementations that implement the server-side chat and matchmaking logic supported by Battle.net to varying degrees of accuracy. One can today download a third party server that emulates the Battle.net protocol, and a third party client that emulates a Blizzard client supporting the Battle.net protocol, and have the two inter-operate. 3) Battle.net issues By virtue of supporting so many different game clients (at present, there are 11 distinct Blizzard-supported programs that connect to Battle.net), Blizzard has a sizable version-control problem. In fact, this problem is compounded by several issues. First, many client game patches add or change the protocol in significant ways. For instance, the notion of password-protected, persistent player accounts was not originally even designed into Battle.net, and was added at a later date via a client patch (and server-side modifications). On top of that, many clients also have very significant differences in feature support. To give an example, for many years Diablo and Diablo Shareware were both supported on Battle.net concurrently while Diablo supported user accounts and the shareware version did not. As one can imagine, this sort of thing can give rise to a great many problems. The version control and update mechanism is not separate from the rest of the protocol. Indeed, the same server, and the same connection, are used for version control, but a different connection to the same server is used for the transfer of client patches. As a result, any compliant Battle.net server is required to support not only the current Battle.net protocol version that is in use by the current patch level of every existing client, but it must also support the first few messages used by every single version of every single Battle.net client ever released, or at least until the version checking mechanism can be invoked to distribute a new version (which is not the first task that occurs in some older iterations of the protocol). To make matters worse, there is now a proliferation of third party clients using the Battle.net protocol (to varying degrees of accuracy compared to the Blizzard game clients they attempt to emulate) in use on Battle.net today. This began sometime in mid-1999 when a program called ``NBBot'',authored by Andreas Hansson, who often goes by the handle ``Adron'', entered widespread distribution, though this was not the intent of the author. NBBot was the first third party client to emulate the Battle.net protocol to an extent that allowed it to masquerade as a game client. Several years later, the source code for this program was inadvertently released to wide-spread public distribution, which kicked off large-scale development of third party Battle.net protocol clients by a number of authors. Despite all of these challenges, Blizzard has managed to keep Battle.net up and running for nearly a decade now, and claims over a million active users. However, the road leading up to the present day has not been ``clear sailing'' for Blizzard. This leads us into some of the specific problems facing Battle.net leading up until the present day. One of the major classes of problems encountered by Blizzard as Battle.net has grown is that it was (in the author's opinion) simply not designed to support the circumstances in which it eventually ended up being used. This is evident in a variety of events that have occurred over the past few years: - The addition of persistent player accounts to the system. - The addition of the text-based chat protocol to the system. - Significant changes to the backend architecture utilized by Battle.net. Although it is difficult to provide exact details of these changes, having not worked at Blizzard, many of them can be inferred. 3.1) Network issues Battle.net was originally setup as a small number of linked servers placed at various strategic geographical locations. They were ``linked'' in the sense that players on one server could interact with players on a different server as seamlessly as with players connected to the same server. This architecture eventually proved unsupportable, as increasing usage of Battle.net led to the common occurrence of "server splits", in which one or more servers would be unable to keep up with the rest of the network and become temporarily disconnected. Eventually, the system was split into two separate networks (each starting with a copy of all account and player data present at the time of the division): The Asian network, and United States and European network. Each network was comprised of a number of different servers that players could connect to in an optimized fashion based on server response time. Some time later, even this system proved untenable. The network was once again permanently fragmented, this time splitting the United States and European network into three subnetworks. This is the topology retained today, with the networks designated ``USEast'', ``USWest'', ``Europe'', ``Asia''. It is believed that all servers in a server network (also referred to as a ``cluster'' or ``gateway'') are, at present, located at the same physical hosting facility on a high-speed LAN. As new game requirements came about, a new architecture for Diablo II and Warcraft III as required. In these cases, games are hosted on Blizzard-operated servers and not on client machines in order to make them more resilient from attempts to hack the game to gain an unfair advantage. There are significant differences to how this is implemented for Diablo II and Warcraft III, and it is not used for certain types of games in Warcraft III . This resulted in a significant change to the way the service performs it's primary function, that is, game matchmaking. 3.2) Client/Server issues Aside from the basic network design issues, other problems have arisen from the fact that Blizzard did not expect, or intend for, third party programs to use its Battle.net protocol. As a result, proper validation has not always been in place for certain conditions that would not be generated through the Blizzard client software. As mentioned earlier, many developers eventually turned to the using the Battle.net protocol directly as opposed to the text-based protocol in order to circumvent certain limitations in the text-based protocol. There are a number of reasons for this. Historically, clients utilizing the Battle.net protocol have been able to enter channels that are already full (private channels on Battle.net have a limit of 40 users, normally), and have been able to perform various account management functions (such as creating accounts, changing passwords, managing user profile information, and so-forth) that are not doable through the text-based protocol. In addition to having access to extended protocol-level functionality, clients using the Battle.net protocol are permitted to open up to eight connections to a single Battle.net network per IP address (as opposed to the text-based protocol, which only allows a single connection per IP address). This limit was originally four connections per IP address, and was raised after NATs, particularly in cyber cafes, gained popularity. This was particularly attractive to a number of persons on Battle.net who used third-party chat clients for a variety of reasons. The primary reason was generally the same ``channel war'' phenomenon that has historically plagued IRC was also rather prevalent on Battle.net, and being able to field a large number of clients per IP address was seen as a significant advantage. Due to the prevalence of ``channel wars'' on Battle.net, artificially large numbers of third-party clients utilizing the Battle.net protocol came into use. Although it is difficult to estimate the exact number of users of such clients, the author has observed upwards of several thousand being logged on to the service at once. The development and usage of said third party clients has resulted in the discovery of a number of other issues with Battle.net. While most of the issues covered here are either already fixed or relatively minor, there is still value in discussing them. 3.2.1) Client connection limits Through the use of certain messages in the Battle.net protocol, it is possible to enter a channel beyond the normal 40 user limit. This was due to the fact that the method a game client would use to return to a chat channel after leaving a game would not properly check the user count. After miscreants exploited this vulnerability to put thousands of users into one channel, which subsequently lead to server crashes, Blizzard finally fixed this vulnerability. 3.2.2) Chat message server overflow The server software often assumed that the client would only perform 'sane' actions, and one of these assumptions dealt with how long of a chat message a client could send. The server apparently copied a chat message indicated by a Battle.net protocol client into a fixed 512-byte buffer without proper length checking, such that a client could crash a server by sending a long enough message. Due to the fact that Blizzard's server binaries are not publicly available, it would not have been easy to exploit this flaw to run arbitrary code on the server. This serious vulnerability was fixed within a day of being reported. 3.2.3) Client authentication Aside from general sanity checks, Blizzard also has had some issues relating to authentication. Blizzard currently has two systems in use for user account password authentication. In order to create a third party client, these systems had to be understood and third party implementations reduced. This has revealed several flaws in their implementation. The first system Blizzard utilizes is challenge-response system that uses a SHA-1 hash of the client's password. The game client implementation of this system lowercases the entire password string before hashing it, significantly reducing password security. (A third party client could opt not to do this, and as such create an account that is impossible to log on to through the official Blizzard game clients or the text-based protocol. The text-based protocol sends a user's password in cleartext, after which the server lowercases the password and internally compares a hash of it with the account in question's password in a database.) However, a more serious security problem remains: in SHA-1, there are a number of bit rotate left (``ROL'') operations. The Blizzard programmer responsible for implementing this apparently switched the two parameters in every call to ROL. That is, if there was a ``define ROL(a, b) (...)'' macro, the programmer swapped the two arguments. This drastically reduces the security of Battle.net password hashes, as most of the data being hashed ends up being zero bits. Because of the problem of incompatibility with previously created accounts, this system is still in use today. The second system Blizzard utilizes is one based off of SRP (Secure Remote Password, see http://srp.stanford.edu). Only Warcraft III and it's expansion use this system for password authentication. This product has it's own account namespace on Battle.net, so that there are no backwards compatibility issues with the older ``broken SHA-1'' method. It is worth noting that Warcraft III clients and older clients can still communicate via chat, however - the server imposes a namespace decoration to client account names for communication between namespaces, such that a client logged on as Warcraft III would see a user ``User'' logged on as Starcraft on the USEast Battle.net network as ``User@USEast''. However, this system is also flawed, albeit less severely. In particular, the endian-ness of calculations is reversed, but this is not properly accounted for in some parts of the implementation, such that some operations expecting to remove trailing zero bits instead remove leading zero bits after converting a large integer to a flat binary buffer. There is a second flaw, as well, although it does not negatively impact the security of the client: In some of the conversions from big numbers to flat buffers, the server does not properly zero out bytes if the big number does not occupy 32 non-zero bytes, and instead leaves uninitialized data in them. The result is that some authentication attempts will randomly fail. As far as the author knows, this bug is still present in Battle.net. 3.2.4) Client namespace spoofing With the release of Warcraft III, a separate account namespace was provided for users of that product, as mentioned above. The server internally keeps track of a user's account name as ``xusername'', where x is a digit specifying an alternate namespace (the only currently known namespace designation is 'w', for Warcraft III). This is known due to a message that exposes the internal unique name for a user to protocol clients. While the character '' has never been permitted in account names, if a user logs on to the same account more than once, they are assigned a unique name of the format 'accountnameserial', where 'serial' is a number that is incremented according to how many duplicate logons of the same account there are. Due to a lack of parameter checking in the account creation process, it was at one time possible to create accounts,via a third party client, that were one character long (all of the official game clients do not allow the user to do this). For some time, such accounts confused the server into thinking that a user was actually on a different (non-existent) namespace, and thus allowed a user who logged on to a single character account more than once to become impossible to 'target' via any of the user management functions. For example, such a user could not be sent a private message, ignored, banned or kicked from a channel, or otherwise affected by any other commands that operate on a specific user. This was, of course, frequently abused to spam individuals with the victims being unable to stop the spammer (or even ignore them!). This problem has been fixed in the current server version. 3.2.5) Username collisions As referred to in the previuos sub-section, for some time the server allowed Diablo Shareware clients. These clients did not log on to accounts, and instead simply assigned themselves a username. Normal procedures were followed if the username was already in use, which involved appending a serial number to the end to make a unique name. Besides the obvious problem of being able to impersonate someone to a user who was not clever enough to check what game type one was logged on as, this creates an additional vulnerability that was heavily exploited in ``channel wars''. If a server became split from the rest of the network due to load, one could log on to that server using Diablo Shareware, and pick the same name as someone logged on to the rest of the network using a different game type. When the server split was resolved, the server would notice that there were now two users with the same unique name, and disconnect both of them with the ``Duplicate username detected.'' message (this is synonymous with the ``colliding'' exploits of old that used to plague IRC). This could be used to force users offline any time a server split occurred. Being able to do so was desirable in the sense that there could normally only be one channel operator in a channel at a time (barring server splits, which could be used to create a second operator if the channel was entirely emptied and then recreated on the split server). When that operator left, the next person in line would be gifted with operator permissions (unless the operator had explicitly 'designated' a new heir for operator permissions). So, one could ``take over'' a channel by systematically disconnecting those ``ahead of'' one's client in a channel. A channel is ordered by a user's age in the channel. 3.2.6) Server de-synchronization At one time, a race condition such that if a malicious user were to log on to two connected (i.e. not-split) servers at the same time, the two servers would cease to communicate with another, causing a server split to occur. It is difficult to provide an exact explanation for why this would occur given the collision elimination mechanism described above for users that are logged on with the same unique name, but it is assumed that in the process of synchronizing a new user between servers, there is a period of time where that a second server can also attempt to synchronize the same user and cause one of the servers to get into a invalid state. According to observations, this invalid state would eventually be resolved automatically, usually after 10-15 minutes. 3.2.7) Seeing invisible users Battle.net administrators have the ability to become invisible to normal users. However, until recently, this was flawed in that the server would expose the existence of an invisible user to regular users during certain operations. In particular, if one ignores or unignores a user, the server will re-send the state of all users that are ignored or unignored in the current channel. Before this bug was fixed, this list included any invisible users. It is worth noting that the official game clients will ignore any unknown users returned in the state update message, so this vulnerability could only be utilized by a third party client. 3.2.8) Administrative command discovery Originally, Battle.net would provide no acknowledgement if one issued an unrecognized chat command ("slash-command"). Blizzard later changed the server software to respond with an error message if a user sent an unknown command, but the server originally silently ignored the command if the user issued a privileged (administrator-only) command. This allowed end users to discover the names of various commands accessible to system administrators. 3.2.9) Gaining administrative privileges Due to an oversight in the way administrator permissions are assigned to Battle.net accounts, it was at one time possible to overwrite the account of an administrator with a new account and keep the special permissions otherwise associated with the account. (An account can be overwritten like so if it has not been accessed in 90 days). This could have very nearly resulted in a disaster for Blizzard, had a more malicious user discovered this vulnerability and abused such privileges. 3.2.10) Obtaining passwords Eventually, Blizzard implemented a password recovery mechanism whereby one could associate an e-mail address with an account, and request a password change through the Battle.net protocol for an account at logon time. This would result in an e-mail being dispatched to the registered address. If the user then replied to the mail as instructed, they would be automatically mailed back with a new account password. Unfortunately, as originally implemented, this system did not properly perform validation on the confirmation mail that the user was required to send. In particular, if a malicious user created an account ``victim'' on one Battle.net network, such as the Asian network, and then requested a password reset for that account, they could alter the return email slightly and actually reset the password for the account ``victim'' on a different Battle.net network, such as the USEast network. This exploit was actually publicly disclosed and saw over a day of heavy abuse before Blizzard managed to patch it. 4) Battle.net server emulation Blizzard 'declared war' on the programmers of servers that implement the Battle.net protocol some time ago when they took the developers of ``bnetd'' to court. As of Warcraft III, they have taken active measures to make life difficult for developers programming third party Battle.net-compatible servers. In particular, two actions are of note: During the Warcraft III Expansion beta test, Blizzard implemented an encryption scheme for the Battle.net protocol (this was only used during the beta test and not on production Battle.net). This consisted of using the RC4 cipher to encrypt messages send and received from the server. The tricky part was that Blizzard had hardcoded constants that were encrypted using the cipher state, but never actually sent on the wire (these constants were different for each message). This made implementing a server difficult, as one had to find each magic constant. Unfortunately, Blizzard neglected to consider the policy of someone releasing a hacked version of the client that zeroed the RC4 initialization parameters, such that the entire encrypted stream became plaintext. After several patches, Blizzard implemented a scheme by which a Warcraft III client could verify that it was indeed connecting to a genuine Blizzard Battle.net server. This scheme worked by having the Battle.net server sign it's IP address and send the resulting signature to the client, which would refuse to log on if the server's IP address did not match the signature. However, in the original implementation, the game client only checked the first four bytes of the signed data, and did not validate the remaining (normally zero) 124 bytes. This allows one to easily brute-force a signature that has a designed IP address, as one only has to check 32 bits of possible signatures at most to find it. 5) Conclusion Developing a platform to support a diverse set of requirements such as Battle.net is certainly no easy task. Though the original design could have perhaps been improved upon, it is the author's opinion that given what they had to work with, Blizzard did a reasonable job of ensuring that the service they set out to create stood the test of time, especially considering that support for all the future features of their later game clients could not have been predicted at the time the system was originally created. Nevertheless, it is the author's opinion that a system designed where clients are untrusted and all actions performed by them are subject to full validation would have been far more secure from the start, without any of the various problems Blizzard has encountered over the years.