With Twitter being down, and feeling the need to spew out characters to the Twinterweb, I opened up trusty MarsEdit and decided to update my good ol’ blog.

That’s when I realized how much Twitter had taken over.

I hadn’t logged into my web interface in a bit, so I had a WordPress update, 4 plugin updates, and a bunch of pending spam. I cleared that out and then started crafting a post in MarsEdit. As I went to post those fateful words, MarsEdit choked on an XML-RPC error complaining about bad content.

I quickly scanned through recent post, looked for bad characters. I used the Googles to try to find an explanation. Finally, I noticed that each page of my blog was spitting out a PHP warning (because, you know, PHP is pretty dumb, and I’m too lazy to have turned off warnings) that it couldn’t download my latest tweets from Twitter.

One quick click of “Disable” and MarsEdit sprung back to life.

Twitter was breaking my site.

At which point, it dawned on me, that at this point, Twitter being down is like having part of the internet’s routing being down. It’s tied into so many systems, and so much traffic/content flows through it, that when it goes kablooey, all that content has to route elsewhere. Which then floods those systems, and they start to struggle and burst at the seams a bit, and then folks overflow into another system, and so on.

Until, the end result, of course, is that Twitter was breaking my site.

Here’s a graphical representation:

It’s fixed now.

My site, that is, not Twitter, which is still down.

HTTP Client – Mac Developer Tool for HTTP Debugging

If you do any web development, this tool is awesome.

My company is at the beginning of a what will end up being a fairly long, exhaustive migration process. Probably on the order of 12-16 months, migrating web sites from a set of servers on one side of the country to a set of servers on the other side. It’s not your typical forklift migration (where you actually move the servers and plug them in at their new home); instead, it’s literally moving files, mail, DNS, etc. to a new platform.

It’s pretty daunting, pretty complicated, and can occasionally be pretty cool.

On the flip side, it’s now 2:55AM Eastern in Boston (where I started my day), but I’m in Phoenix where it’s actually only 11:55PM. That’s a sign that maybe things didn’t go quite as smoothly as one would have hoped.

The step we’re on is a step where we take over DNS for folks. It’s always somewhat difficult, because we’ll get a big list of domains and have to figure out whose record (our nameserver’s or the other nameserver’s) is the “real” record. It’s not generally too tough to figure it out (you can judge by the SOA of the records) and the number of domains is usually short of 100k, so as long as you’re accurate to within 1-2%, it’s not too bad. That’s 1000 guys who might break, which is pretty easy to handle with a good support team and some quick script fixes.

Let me take a step back. The process is actually that our nameservers need to become the authoritative nameservers for the domains we’re moving. This allows us to later change their DNS to point to their new home, and it all kinda works. We have to get the domains, merge them into our nameservers, become authoritative, and then fix what breaks.

We did that yesterday. Except it wasn’t 100k domains. It was 1.2 million. And the domains weren’t coming from a single, well-maintained nameserver. They domains came from three, somewhat munged together nameservers. There were internal conflicts, conflicts with our servers, missing zones. A host of issues. We thought we’d worked most of them out and gotten the problems down to, at most, 4-5k domains. That’s a lot, but in reality, it’s less than 0.5% of the total domains.

“Pretty good,” you say.

“Not quite,” I say.

For you see, there weren’t just three nameservers. There were five. So there’s a couple thousand domains we missed. And we also missed some of the conflicts (either by omission or by grabbing the wrong data). In the end, it was closer to 12k domains that were wonky.

That’s still only 1%. Damn good, given all of the variables.

Except 12000 broken domains leads to a whole lot of phone calls and emails. And some angry customers. And some tired folks staying up to fix things that they weren’t responsible for breaking. And one tired folk–me–staying up because he feels guilty for only being 99% good enough.

DNS is a fickle beast. Thankfully, it’s pretty quickly fixable. Once we’d identified some global problems, we could fix them rapidly and put big chunks of the broken domains back in working order.

I often argue with people who think the “Chinese Market” is a valid business plan. You know, the folks who say “hey, if we can just get our product in front of 100 million people, and get 1% of those people to buy, we’ll be rich!” Except, of course, it doesn’t really work that way. It’s hard to get a product in front of that many people who would be interested in buying, and it’s hard to get 1% of any audience to buy anything.

Well, not in business plans, at least. It does work that way in technical issues. If you’ve got a huge enough base of users, the smallest mistakes can have a big impact on your company and team. In these cases, sometimes being 99% accurate isn’t good enough.

Here’s a graphical representation:

You see, with 100k domains, you never quite reach screwed. It’s manageable.

With 1.2 million domains, you’re pretty much totally screwed.

I poked over to Slashdot this morning before I ran off to basketball. I saw this:

Holy crap. That’s ugly.

Clicking the little link to vote takes you to a Thank You screen that says that voting will help ensure the best stories are presented. I would have expected something more interactive, more (dare I say it) Web 2.0-y.

Then I actually read the Thank You message:

“Thank You for participating in the Slashdottit Rating System

Your vote in the Slashdottit system will help insure the best stories are presented to our readers, with hyper accurate numbers to indicate their relevance and general awesomeness. Only by requiring the contribution of every random user of the internet can we guarantee the most scientifically perfect numbers will be generated. These numbers will guarantee that every story that appears on Slashdot will be interesting, insightful, and flawless. Vote early, vote often.”

Ha. Clever. April Fool’s. Well played.

I’ve been reading a lot of search engine stuff in my feed reader recently. I used to be deep into the search engine optimization knowledge, but at some point, I realized that it was, at some level, just scummy. Not the idea that you’d understand how engines work and do the little things to make your site rank appropriately. No, it was the other stuff, like link exchanges and link buying and the general dishonesty that comes along with that. When I go to a search engine, I want to actually find what I’m looking for, not have to dig through a bunch of crappy sites that think they deserve my traffic.

It got worse when AdSense came along, and it got even worse as Digg, Facebook, MySpace, and the other social networking-type sites got big. Now, not only were people gaming the engines, they were throwing up lame articles and gaming other systems to get both the search juice and the traffic. Their spammy site gets the best of both worlds, and the rest of us deal with more spam–just not of the email variety.

This week was a big to-do about one of these SEO/SMO guys who got banned from now-Yahoo! owned blog widget because he was posting how to hack it (and, quite frankly, being an all-around douche). So, a guy who games the system for a living was bitching about being banned from a free tool that he’d been posting how to hack. Topping it off, a whole bunch of other SEO folks (many of whom I’ve been reading for a few years now) hopped on and defended the guy.

I just don’t get it.

I understand that the whole idea behind this widget (MyBlogLog) and behind other sites (like Digg, Flickr, etc.) is community. You build a community and you get more than just the functionality of the widget, you get the benefit/fun of the community. It’s all so Webtwopointohy.

Finally, a voice of reason came through my feed reader. I’m hoping we’re reaching a tipping point. I’m hoping we’re reaching the point where every sales and marketing guy out there looking to score some quick money doesn’t look at every new site and widget as something to game and make money. Now, I’m not against making money. I’d love to create a site that has some value to people and figure out a way (ads or not) to make some money. But the group of folks who exist solely to put up a site with ads, get it on Digg, and get enough sheep to click on it need to go away. They used to be called spammers, and it’s about time we go back to calling them that.