Wells Fargo: WTF?
August 22nd, 2007I’d really like to know more about the nature of what actually happened with this Wells Fargo disaster.
Redundancy is one of the main characteristics of mission critical systems. Wipe one box off the face of the earth, and there could be one, ten or dozens more, in different locations, waiting to take up the slack—and without even skipping a beat. It is incomprehensible that Wells Fargo’s core transaction processing capabilities were reliant on a single set of servers.
Wells Fargo said that they had to recover from tape, but WTF happened to the redundant nodes?
Via: Computer World:
Wells Fargo & Co. customers may have a hard time getting an up-to-date balance statement today, as the nation’s fifth-largest bank continues to iron out service problems related to a Sunday computer failure.
The outage knocked the company’s Internet, telephone and ATM banking services offline for several hours, and Wells Fargo customers continued to experience problems today.
Wells Fargo didn’t offer many details about the system failure, but it was serious enough that the company had to restore from backup.
“Using our backup facilities, we restored Internet banking service in about one hour and 40 minutes,” the company said in a statement today. “We thank the hundreds of team members in our technology group for working so hard to resolve this problem.”
Other banking services such as point-of-sale transactions, loan processing and wire transfers were also affected by the outage, and while all systems are now fully operational, some customers may continue to see their Friday bank balances until the end of the day, Wells Fargo said.
Customers couldn’t perform transactions during the outage, but the company’s Web site was not knocked offline, according to data from Netcraft Ltd., a Bath, England-based Web analysis firm.
“We apologize for the inconvenience this caused,” the Wells Fargo statement said. The company did not return a telephone message seeking additional information about the outage.
As I said earlier, it’s a test run. “What if we had to prevent a bank run on very short notice? What are the ramifications? How long would it take to bring us back up? Who in the organization would we have to take out of the loop and/or reassign so they wouldn’t become suspicious or worse?” Etc.
Reminds me of your post about how easily the whole net could go down from targeting a few critical points. Scary stuff..
I’m with you. It seems inconceivable that a large banking network could be brought to its knees by a server failure.
It’s been awhile since I studied networking or had employment-related dealings with this type of thing, but big companies are normally freaking obsessive about the security and redundancy of their servers.
I think there’s more to this story than anyone is letting on. Though it’s POSSIBLE that this resulted from yet another characteristic of big companies: They have cut every possible corner when it comes to employing IT people. Tech support people used to complain that problem software was the result of employing only temporary and seasonal programmers. Maybe they’ve all gone to contracting out their network administration and are changing network administrators more often then they’re changing their socks.
But I doubt that this is where the real problem lies…and I’ll bet we’ll see more of the same with other banks.
According to Housing Panic today
http://housingpanic.blogspot.com/2007/08/want-to-see-housing-version-of.html
Wells Fargo is the #1 lender (in Q3 ’06) of residential jumbo, subprime, and interest only mortgages. They even beat out Countrywide Financial, which from what I understand is now on the brink of collapse.
Maybe this is a test run. How convenient would it be, if when the shit hits and the bank run on Wells Fargo starts, they have a cover story ready for why no one can make any withdrawals. Like, oh I don’t know, “some red necks shot up our fiber optic cables?”
And they probably got the inspiration from your essay, Kevin. lol.
Clicking over to the original Computer World article, I notice that the comments express a certain amount of suspicion.
Now that JC Denton mentions it, I do recall that Wells Fargo has been a big mortgage lender over the past few years–and apparently all the big lenders were heavily into the subprime and other “iffy” markets.
The lender who will get their asses kicked are, I suspect, the ones that made loans to the biggest real estate developers–the kinds of developers who buy up a few hundred acres of farmland and put up a few hundred acres of McMansions. This type of thing has been going on for decades in my area. Until now, I’ve never been particularly curious to know who’s bankrolling these schemes–which I do know are often executed using borrowed money.
One of my near neighbors owned a construction company. At the time of 9/11, she had about a dozen houses under construction–all on borrowed money. After 9/11, she was unable to sell ANY of them before the loans came due, and she wound up in bankruptcy–and out of that business permanently. This woman is a dear, charmingly crazy sort of woman–the kind you would never wish any ill. I mention her case by way of illustrating how the development business appears to be run: Take out a multi-million-dollar loan, with no payments and no interest for the next few months, build a bunch of big houses quickly–and pray you can sell them quickly, so you can use the proceeds of the sale to pay off your construction loans.