(Note that he does make me look like a bit of a doof at one point. I have three things to say in response: 1 – I’m still pretty happy with how we managed to get a working CS front-end and back-end system up and running in about six weeks, 2 – it wasn’t *quite* as doof-tastic as Firor makes out, as I explain in annotations to his tale, 3 – I am in fact a bit of a doof.)
And with that, I give you Matt Firor.
We went into October 9 relatively calm and serene. Vivendi, our distribution partner, had forecast 100,000 sales of the game lifetime, with about 50,000 coming in the first couple of days after launch, and as such, they only “sold in” a very limited number of boxes into the retail channel. We were very comfortable that we could handle those numbers, as we had just had a very successful beta program.
Then, just before launch day, Vivendi got in touch with us and said because customer/retail demand was so high, they were going to release all 100,000 boxes into retail. Fortunately, with delivery times, these extra copies would be delivered to stores a couple of days after the initial 50,000 boxes. This was very helpful to us, as all the boxes were not available on the same day which spread out the “opening day” crush of users over five days. Camelot would go on to sell more than a million boxes in the next couple of years.
Camelot’s official launch day (as in boxes were in the stores) would be October 9 2001, but everyone that had a free account (lots of media, some friends/family, and of course all of us) were able to play starting October 8, as soon as we put our first seven servers online starting around 5:00pm that day.
At the time, Mythic’s offices were located in a townhouse office community near the middle of Fairfax, VA. We were about 30 developers in October 2001, and about 40 customer support. We didn’t have enough space for CS in our original office, so we had to lease space in another building in the same complex, about 50 yards from the developer wing of the company, in a basement space.
Everything was done on a shoestring at that time, so to get internet access over to the CS “center” 50 yards away, Rob Denton, the Development Head of Mythic at the time, and an electrical engineer by training, set up a pair of IR “guns” to make a link between the two spaces. We put one “gun” in a window in the development office, and another in a basement window, pointing up, in the CS center. The link worked very well, and allowed us to share our one Internet line with both spaces. However, because the CS Center was below grade, we had one problem: if a vehicle over a certain height (about 5 feet) parked in a particular parking spot, the link would be broken. We lived in fear those couple of weeks that a delivery truck would park in that spot and cut off Internet access to the CS center. We arranged a quasi-official parking schedule to ensure that an employee car (a short one) be parked in that spot 24/7. The link, fortunately, was never broken.
It's on.
All day on the 9th, we watched as the server numbers grew and grew. I ran the login utility on my laptop all day, just so I could see the population numbers of each server, real time. The population numbers started small around 10:00am on the 9th, and grew slowly but steadily until around 5:00pm, when they exploded. All servers in the space of about an hour after 5:00pm were jammed full – and we had a very large server population setting (about 3,000 players). Even with full servers, everything ran smoothly.
By about 8:00 we were jubilant. Everything was smooth and easy. CS was functioning, and had already responded to many trouble tickets and issues. People were playing, the servers were up.
A group of us formed in Rob’s office, talking and generally basking in the glory of the moment. Each of those 30,000 (max peak players that night) was a paying customer, and each represented significant revenue to us (remember we were very small at the time). It appeared that we finally were going to make money on one of our products. We were giddy with excitement – everything was going awesomely.
Brian Axelson, the 21 year old whiz-kid programmer/designer who had been working for us since he was 16 – responsible for inventing, implementing, and designing Camelot’s combat system, including Combat Styles – was so happy he slammed his fist down on Rob’s desk and said, “Ain’t nothing going to bring this house down!”.
At that moment, all the servers crashed, simultaneously.
We all looked at one another in dread, and sprinted back to our offices, each checking on the part of the game we were responsible for. Everything checked out – nothing seemed wrong. But the servers were down and wouldn’t reboot.
All the programmers were summoned to Rob’s office – I was a fly on the wall – and he walked them through the problem. It wasn’t a code problem, although that wasn’t immediately obvious. It was something keeping the servers from booting and authenticating properly.
After about an hour, the problem was traced to the Customer Support tool – that very day programmer Scott Jennings had made a small modification to the CS tool to take advantage of a database feature buried in MySQL to make database queries work faster. That change did in fact significantly increase the speed at which the tool made queries to the database – but at full load, the index that he built quickly became overloaded, and started to time out and lock out other queries. (Editor’s note: This change was actually fully tested… with one low-population test server running. Guess what changed on launch day!) Because the game servers relied on access to the database as well (for player authentication, etc.), they couldn’t keep up with the crush of players logging on and off – and they crashed like the proverbial house of cards. And, because the database was locked up, when they rebooted, they immediately ran into the same problem and crashed again.
The helpful message I added to greet CS reps on launch day.
Once the problem was found, it was very easy to fix, for the moment. The CS tool was modified to not make any of those specific types of queries, the database server was rebooted, the index rebuilt, and everything came up again – this time smoothly and without error – and ran flawlessly until the next afternoon, when we had our first bug-fix patch. Scott had fixed the DB/authentication problem in the meantime (Editor’s note: and said programmer slept three days later), and that functioned properly as well.
In the end, a very smooth launch, but a lesson was definitely learned that when you’re dealing with something as complex as a MMOG launch, you never know what is going to take you down.
There are many more stories to tell about the early days of the service – like how we had to expand servers quickly because of demand, but couldn’t get them delivered from Dell because we had no credit rating. All our purchases up until that point had been made on the spot with no leasing. We had no leasing history, so Dell wouldn’t ship us servers quickly. We were forced to drive to MicroCenter (in Fairfax) and buy a dozen or so desktops, quickly installed Linux, and then drove them (in a pickup truck) to our colocation facility, and stacked them up like firewood in a cage. Those two servers clusters (lovingly called the “gimp servers”) ran for at least a year with no problems, at which point they were swapped out with standard Dell rack-mounted models.
I remember walking into the office one morning towards the end of October. By that time it was obvious we had a smash hit on our hands. Our marketing/sales consultant, Eugene Evans (now the GM of the studio) had a whiteboard near his desk (right by the front door) where he jotted down sales numbers. By October 27 or so, it showed that we were not only the #1 selling PC game for October, but also the #1 selling PC product for that month. Since this was the first boxed retail product Mythic had, I asked him if this success was normal. Eugene, and old industry veteran, looked at me like I was insane and replied, “No, this isn’t typical.” He then broke out in laughter. It seemed so easy at the time: you make a game, put it in a box, and it sells like hotcakes.
Most everyone knows the rest of the story from here – Mythic quickly outgrew its space and in 2002 relocated a few miles away to new mid-rise building, where it grew to take over three floors. A buyout by EA followed in 2006, and the studio is now known as Bioware Mythic. Dark Age of Camelot’s numbers have dwindled down to a fraction of what they were in those heady days of 2001-2003, but it is still up and running, ten years later.
Read Full Article
