Archive for the ‘disaster recovery’ Category

Web 2.0 – Code4Lib addresses Data Mgmt – When will You?

July 30, 2007

As you know, I’ve been concerned about the institutions that host data for Web 2.0 applications. Code4lib, a major library 2.0 site (and everything else hosted on anvil.lisforge.net) was hacked on July 21 and is still not available. They are hoping to have everything back on Aug 1 – we’ll see.

And 6 back-to-back power outages hit the SOMA neighborhood of San Francisco last Tuesday afternoon causing major havoc with popular web services. 365 Main was down, along with craigslist, Technorati, Yelp, AdBrite and SixApart (including TypePad, LiveJournal and Vox). Many other popular sites such as CNet were unavailable too.

I wrote a couple of posts about these problems and suggested that it is is greater issue earlier – this one on the 365 Main Outage and some thoughts and this one on if you trust online sites to protect your data re: Code4lib.

Well, Code4lib is taking this seriously (as they certainly should) and is hosting a special discussion on August 1st to discuss this. Here is the announcement from their Planet Code4lib website (the only code4lib site currently available).

“You are invited to a special discussion in #code4lib on irc.freenode.net on 1 August 2007 at 1900 GMT about how to prevent this from happening again. We’re going to be talking about moving some of the web applications to institutions that are better set up to manage them.”

I am thrilled that code4lib is now thinking about this and I hope they can recover all their data in a timely manner. And I hope that other organizations that are heavily web-based will follow their lead and seriously look at who is hosting their data and that they are thinking about ensuring that they know what is in place to protect them.

In the Web 2.0 world, it isn’t just about content and collaboration and new ways to interact. Now that these Web 2.0 concepts are coming to fruition and are becoming valuable resources, it is time to look at making sure they are operating in a stable and protected environment.

~Susan Mellott

Major Web 2.0 Sites Down from Power Outage – They need a lesson from Big Business.

July 26, 2007

Power Outage in SF Tuesday brought down major Web 2.0 sites.

6 back-to-back power outages hit the SOMA neighborhood of San Francisco Tuesday afternoon causing major havoc with popular web services. 365 Main is down, along with craigslist, Technorati, Yelp, AdBrite and SixApart (including TypePad, LiveJournal and Vox). Many other popular sites such as CNet were unavailable too.

Interestingly enough, a “source close to the company” (365 Main) had this to say:

“Someone came in sh*tfaced drunk, got angry, went berserk, and f**ked up a lot of stuff. There’s an outage on 40 or so racks at minimum.” ValleyWag had a good article on this with lots of interesting links.

This however was unlikely as the cause since the area had been having power outages and clearly their UPS system did not function properly.

The San Francisco website Laughing Squid has a write-up of the power outage .

Here is another informative post from Radar.OReilly.com

Six Apart, in a very 2.0 move, kept everyone updated via its twitter stream.

But the real question is, what happened to their power backups? They should be able to keep running regardless of any lack of power. This is a good post about what 365 had to say regarding its “Credibility Outage” (and basically they made a bunch of excuses).

So again, do you trust your Web 2.0 online providers? Clearly there is a gap between what “should” have happened and what actually did happen. Datacenter 365 Main released a self-congratulatory announcement celebrating two years of continuous uptime for client RedEnvelope, mere hours before today’s drunken blackout.. [PR Newswire]

And without extensive testing and backout plans, it is hard to know what exactly would happen if something happened like a server being hacked or a major power outage. I would be more interested in the disaster recovery plans and testing they (or any major player) had done than in what they theoretically think might happen, based on the things they think they have in place.

Coming from a big business background, where their only real commodity is data (in my case, insurance), I have seen and been involved in a huge amount of disaster recovery testing and planning. I remember what they, and other businesses went through for testing for the 2000 rollover and for any number of other potential disasters. September 11th tested their and many others disaster recovery plans. The Stock Market and major banks and other financial firms simply cannot just go down or lose data, for any reason.

But as we move to a Web 2.0 world, companies like 365 Main are now also the repositories of major amounts of data and for many Web 2.0 companies, their business is data, just like financial institutions. It’s not small potatoes anymore. Face it, they are big business now and need to act like a big business. I’m sure they are bringing in big business income. So who holds them accountable? I’m wondering if many of these Web 2.0 companies didn’t grow from such small beginnings that they may not even be aware of what they need to ask and know from their provider.

And unfortunately, I hear people with a business background being dismissed as “luddites” or “1.0” or “dinosaurs” or just not with it, supposedly not able to comprehend the new 2.0 world. It reminds me of when PCs first came out and I started programming them after having had a mainframe background for several years.

PC programming was wild and wooly. There were no standards, no one documented their code so maintaining it was a nightmare, and people would see how many functions they could put on 1 line of code (more being better in their mind). An “elegant” piece of code would be completely undecipherable by anyone (which seemed almost to be the point) and would have no documentation. Which meant of course, that the code for most of the programs were a mess because no one could figure out what the last person did so they hacked around it. But if you were from a mainframe background, you supposedly could not “understand” PCs and were basically a dinosaur. Well, I know that is a bunch of nonsense because I didn’t have any problem understanding PCs and PC coding. What I didn’t understand was why they allowed projects and programs to be so sloppy and poorly run and written.

It was a real case of 1.0 technology meets 2.0 technology. In this case, Mainframes vs. PCs. Now it is happening again with Web 2.0. And regardless of what the current “New Thing” ™ is, one thing they all have in common is the belief that they know more than the people who have used the ‘old’ technology. But what they don’t realize is that they really haven’t learned anything at all yet. They have a great direction and new ideas and concepts and great plans, but if for no other reason than that the technology has not been around that long, they don’t have practical experience and a background to build on. I’m sorry, but while college gives you an in and a piece of paper to say you are somebody, the real learning starts when you start applying the knowledge in real world situations.

I remember taking a LOMA (life office management) test on data processing and thinking it should be a piece of cake. It turned out to be one of the hardest of the set because I had to learn what they thought the right answers were, not what was actually correct. I had the same experiences in higher education where what was being taught was so outdated that it was really completely wrong and in my opinion, was harmful in many ways to learn, especially if you thought you knew something afterwards.

And this is where I think the 2.0 arrogance is showing. It is a wonderful new way of doing things, but there are many foundations they could and should build on that have already been figured out. They can take what has been done to new and exciting levels, but reinventing the wheel for every single thing is pointless and causes the new technology to be without wheels for a while.

~Susan Mellott