MyDeadSpace


MySpace went offline for about 12 hours yesterday, reportedly due to the sweltering heat wave ripping through California (I heard yesterday on NPR that there may be some relief thanks to a fog bank coming into San Francisco, but come on, that's only going to benefit SF and its nearby communities, not any cities further south in the Santa Clara Valley).

For nearly 12 hours on Monday, the popular social-networking site, which recently topped Yahoo Mail as the most-visited Web site in the US, was disabled entirely. Its front page was replaced by a message from founder Tom Anderson about fixing a power outage that would optimally be solved within the hour, as well as a Flash game of Pac-Man as a peace offering for disgruntled users.
MySpace returned hours later -- the Associated Press estimated the entire length of the failure at 12 hours -- with scattered complaints continuing throughout the day from individuals who claimed they still couldn't access the site.
A statement from MySpace blamed the outage entirely on the sweltering temperatures that over the weekend crippled power systems throughout California, including the Los Angeles area, home to MySpace's headquarters, where temperatures climbed as high as 48 degrees in some places. "The area where MySpace's servers are stored had massive power outages and the backup generator failed," a company representative explained. "With power resumed, the network is now up and running."

Obviously, someone in the Operations team is going to lose his job for failing to come up with an effective Disaster Recovery plan. It looks like there were at least two consecutive fail points:
- Power went out in the first place
- Backup generator also failed
Did DR consist of a backup generator? Did they assume that power would return in two hours or so, and actually ran out of diesel for the generator, causing the remaining 12 hours of failure? Many of the call centers that I have looked at actually have their power generators connected directly to a natural gas pipeline, so there is no chance of running the tank out before the main power source is restored...

My other thought, though, is how this could accelerate the movement to locate server farms in the Northwest:

There is cheap electricity here, and lots of it. That is because the Columbia, the premier hydroelectric river in North America, flows nearby. Three publicly owned local utilities own five large dams on the river, and they produce much more electricity than the sparse local population can use. With power prices soaring, the three utilities have become the hydroelectric emirates of the Pacific Northwest.
Until now, they have been obligated under 50-year-old contracts to sell about two-thirds of their power — without profit — to major utilities serving millions of people in Seattle, Tacoma and Portland. The arrangement helped keep monthly electric bills in the Northwest far below the national average.
Those old contracts, though, are expiring — a development that will help push up residential electricity rates across the region. And the mid-Columbia utilities are scurrying to sell their newly unleashed power to the corporate giants of the Internet — if they are willing to plant "server farms" in two-stoplight towns such as Quincy.
They do seem uncommonly eager.

Posted: Tue - July 25, 2006 at 08:18 AM        

©