That Not So Fresh Feeling

If you’ve been following along at home, you’ll have noticed I work for a web hosting company. We’re pretty big, with hundreds of thousands of customers, and we’ve got some interesting operation efficiencies/differences from traditional hosts that give us some fun advantages. Those same advantages come at the cost of having to do some interesting thinking about scalability and performance. This is where the stuff currently going on at Twitter starts to sound *really* familiar.

But, let’s start with the basics …

Typical Web Host
A typical web host has a “pizza box” architecture. They pile a bunch of servers in a rack in a data center. Each server is running some MySQL, some Apache (or ISS), some email app (Exim or Qmail), and some FTP. Each server is usually running another instance of Apache to run the customer control panel. The box usually runs off it’s own local storage. Thus, each box is an individual entity running it’s own versions of applications, with it’s own individual issues. Depending on which box you’re on, you might have MySQL slowness, or Apache slowness, or disk slowness, all depending on what your neighbors are doing. Rolling out upgrades becomes a bit more arduous as you’ve got to upgrade each box. This is decidedly old-school, but very common and works fairly well. This architecture only starts to be a problem as you grow; each time you add X customers (where X is the number each box can support), you have to add a new box. Each time you add a new box, you need more power, space, backup storage, etc. That gets hard to scale.

So, then there was the idea of not using local box storage, but instead using networked storage. Now you can just add more disks without having to add entirely new boxes, and backups become less of an issue. This means you can grow a little bit more without having to add more boxes.

Atypical Web Host
An atypical web host (like us and some of our competitors) might do things a bit different. For instance, rather than having a single LAMP (Linux-Apache-MySQL-PHP/Perl/Python/(P)Ruby) stack per set of customers (on a box), you build a pool of boxes that can all serve up services for a customer. This means that:

• You’ve got some redundancy
• You don’t have to scale all of your services as you add new customers

In this model, more like a typical web service than a web host, you’ve got pools of servers that perform certain tasks, and customer data is kind of abstracted away to all just live on big storage arrays. Things scale much easier and more cost effectively.

But ….

What does this have to do with Twitter?

See, when you’re running centralized services, you end up with a lot of data sitting in your database, and it’s getting accessed by Y thousands (tens of, hundreds of) users. The “pizza box” model doesn’t have this issue since the data is distributed across thousands of servers. But that’s not feasible in something like Twitter. You don’t only want to be able to see tweets from people who are on the same server as you. You want to see tweets from everybody. That’s the power of Twitter.

Having lots of data in a single database instance solves that problem. But it introduces a whole new set of issues: database locking, slave synchronization, disk performance.

Let’s take an educated guess at what Twitter’s database schema might be like:

Table: Users
UserID     int(14) autoincrement
Username   varchar(50)

Table: Tweets
TweetID	   int(14) autoincrement
UserID     int(14)
Tweet      varchar(140)
DateStamp  datetime

Table: Followers
UserID     int(14)
FollowerID int(14)

Table: UserTweets
UserID     int(14)
TweetID    int(14)

In a nutshell, we’ve got a table that keeps track of our users. We’ve got a table of tweets, which maps back to give us a user based off of the UserID. Right there, you get a pretty easy query to get all of the tweets for a user:

SELECT t.Tweet FROM Tweets t INNER JOIN Users u USING (UserID);

When Twitter was just a baby, that query’s pretty darn fast. It’ll give you all of the Tweets written by any user in particular. Reads will be fast, since UserID will be indexed and unique. Writes will be fast, since we’re not writing a ton of data and updating the indexes should be pretty quick (since Twitter is still just a baby).

Now, we add the functionality of “following” another Twitter user (good thing we thought ahead and added that to our schema!) Now a user can follow another user, which just simply sticks a row in the table with the id of the user and the id of who they are following.

Keeping track of all the tweets flowing into the user from folks they’re following is easy, since we planned ahead. We made a table that lets us map a bunch of tweets to a user. Every time we add a tweet, we stick a row in that says “this user has this tweet”–it doesn’t matter if the tweet was by that user, or someone they were following, they all go in that mapping table.

Again, when Twitter was a baby, this was all pretty easy and quick:

SELECT t.Tweet FROM Tweets t INNER JOIN UserTweets u USING (TweetID) where UserID = ?;

We’re going to get back all the tweets assigned to that user, whether written by the user or someone they are following. Simple and fast, since again, we’re hitting some pretty easily indexed and unique fields.

Of course, I’ve made one assumption here. I’m assuming that, when a user starts following someone, that their tweets start getting associated to my user (the UserTweets table). I’m guessing this is the case since when you start following someone all of their tweets don’t magically show up in your history (conversely, when you stop following someone, they don’t miraculously disappear — I don’t think so, at least). Either Twitter sticks stuff in the mapping table (my guess), or they go look up the date you started following each user, and they build that index on the fly.

In other words, the alternative version is that Twitter, when you view a page that displays your tweets along with those of folks you follow, would have to go lookup the date when you started following someone, then find all of their tweets after that date, bring them together, and put them in order.

Mapping table seems a whole lot more likely. And much faster. Which is kind of how Twitter got big.

Anyway, since we’ve got this nice mapping table, we need to know how to fill it up. If I’m following Robert Scoble (like the rest of Western Civilization), every time he writes something, it needs to make it to UserTweets associated with me.

Again, this is pretty easy, if you’ve got a handy loop. First, get all the followers:

SELECT FollowerID from Followers where UserID = SCOBLES_ID;

Then, loop through all those folks and post it:

INSERT INTO UserTweets (UserID, TweetID) values (FOLLOWER_ID, SCOBLES_TWEET_ID);

That loop will run as many times as it takes to update Scoble’s followers. None of these queries are complex. They’re all easily indexible. They should all be pretty fast. When Twitter is still in baby-state.

But now Twitter is growing. It’s a toddler. It’s got many many users and some of them have loads of followers. Things are starting to slow down. Twitter’s parents look at it and say “Well, one obvious reason you’re starting to slow down is that the database is starting to lock. See, now that we’ve got some data, lookups are taking a little bit longer, and inserts are taking a little bit longer, and when they happen at the same time, we don’t want data to get out of sync, so the database locks up. When it locks up rapidly enough and often enough, we run out of threads and our database likes to go down. We’re going to teach you to read and write at the same time.”

And that’s what they do.

We currently use one database for writes with multiple slaves for read queries. As many know, replication of MySQL is no easy task, so we’ve brought in MySQL experts to help us with that immediately. (blog.twitter.com)

Now we’ve got a master database, where all of our writes (inserts) go, and a couple of replicas where all of our reads (selects) go. Perfect. Things are fast again. Users are happy. Twitter is moving along.

Twitter grows into the tween stage. And it’s not pretty. Databases are constantly crashing. Things are slow. What the hell? Didn’t we just fix this?

Well, no. We just hid the problem. Replication isn’t a pretty solution. MySQL replication is flawed. It’s not instantaneous. It can fall behind. Further, it breaks. A lot. And when it breaks, we lose half of our read capacity, which then overloads the other server, and everything goes down. Resynchronizing can be painful. Adding more replicas helps, to a point (since each replica adds a little bit of overhead to the master).

Big users (those with lots of followers) cause more load, since now we’ve got to do a big lookup to get thousands of rows (followers) and then do thousands of inserts every time one of those users posts. That’s not good for our database.

NOTE: The Twitter folks claim they don’t copy the tweet around for each follower. I’m sure they don’t. But they don’t say anything about copying the ID around (which, as I’ve stated, makes a good amount of sense if you were architecting Twitter 18 months ago).

Also, it’s what Dare Obasanjo posits, building on what Israel says on the Assetbar blog, and they’re both way smarter than me.

And you can’t really add a second master database. So writes are going to be as fast as our server can handle them.

This is where we are today (or at least recently). There’s enough users on Twitter that updates are probably starting to slow since the tables and indexes are so large that there’s just not much MySQL can do. Reads are slow because replication is fragile.

What do we do?
Without completely re-architecting things, what can we do? There’s a couple of things, right off of my novice brain:

• Cache, Cache, Cache
• Make the database smaller

Cache, Cache, Cache
The less we have to go to the database, the better things perform, and the better they scale. There are lots of things we can cache, things which don’t update very often. For instance, the list of followers. If we stored the follower list in memory for each user (as it was loaded), we’d cut down on a query that gets run every time a user loads Twitter.

That’s *a lot* of queries.

It sounds like the folks at Twitter are already heading down this path.

A: We’ve mitigated much of this issue by using memcached, as many sites do, to minimize our reliance on a database. (blog.twitter.com)

Make the database smaller
This isn’t something it sounds like Twitter wants to try. Basically, this is sharding. You take all the users with a UserID < 100000 and put them in one database, all the users with a UserID > 100000 and < 200000 and put them in a second database, and so on. Then you have a master lookup that lets you know where to find the data you’re looking for (which you can cache!).

This makes your selects and inserts fast again, since the tables and indexes are smaller. But now you’ve got to manage the multiple shards, adjust them when they get too big, and deal with the added layer of abstraction.

It’d help, but it’s a big task.

If you’ve been reading my blog, you might be asking yourself:

Why Does This Sound Familiar?
Why does this sound familiar? Because our unknownish web host ran into similar issues. See my post Thoughts on MySQL Scalability From a Certified MySQL Moron from February.

All of the problems Twitter is facing seem to be directly related to building the service in a logical fashion, but not forseeing the problems that massive growth would have on the system (a single tweet spawning tens of thousands of database updates). Quite frankly, it’s completely reasonable. Frustrating, but reasonable. As Michael Kowalchik (founder of Grazr) states:

As a startup there’s only so much energy you have, and you must apportion your resources carefully. The truth is, we like to talk about scaling, but without steady growth and something people find compelling, all the scaling in the world won’t help you. (mathewingram.com

I’m curious about Twitter’s solutions both as a user and as an engineery web dork at a company hitting the same problems. We’re starting to use memcached, looking at using smarter non-table locking databases, and thinking about utilizing the file system more than a database.

Twitter has some technological mountains to climb to be able to scale to support the rate at which they are growing. I think that, sooner or later, they’ll have to bite the bullet and use database shards. They’re also likely going to have to build out memcached clusters large enough to allow them to cache nearly every thing about a user. That’s loads of data (gigs and gigs and gigs), but machines and memory are cheap for a company like Twitter, and the payoff will be worth it. I have no idea if Twitter’s growth is slowing during these performance issues, but throwing some funds behind a massive memcached cluster would be well spent now, and would surely be useful even after any sort of re-architecture.

I’ve meant to use Skitch for a while, and finally got around to using it today. It’s pretty cool. For instance, you can do a search for “phpbb is a piece” and find some fun links:

Then, if you want, you can do some fun stuff to it. Like add some comments:

Tada!

Awesome. Takes about 2 seconds.

I haven’t been posting very much recently because I’ve sadly been working my tail off. I very much enjoy what I do, but there are just weeks (months … years …) where it’s just a non-stop grind to get everything done. Recently, it’s been working on launching a new VPS platform, but that was interrupted by a breakdown of our customer MySQL infrastructure.

Our setup is a bit different than most. Since we don’t run a typical box-by-box web hosting architecture, we don’t simply have a thousand boxes with each one running Apache and MySQL. Instead, we have a really robust pooled architecture for everything except MySQL, which just isn’t something that’s very poolable. For MySQL, we’ve got some big boxes with a bunch of memory and some fast disks that handle our MySQL load. But, slowly over time, performance had degraded.

When you’d hop onto a box and look at the transactions per second or number of queries, nothing looked terribly out of the ordinary. Yet the load would be huge, and performance would be pretty bad. Our team brought up some new boxes, shuffled customers between them to even the load out, moved our backup processing onto the hot spare replicated boxes (to reduce even more load on the disks) and things were better.

But they weren’t better enough. (I know, awesome English, eh?)

We started just watching the processlist, looking for the culprit. And after about 5 minutes, it was obvious.

Motherfrakking phpBB spam.

phpBB is written in a really shitty way. Not the forum part, necessarily, which works when it’s not being exploited. But the search part is awful. For every word in every post (unless you’ve got a smart list of words to ignore), it throws entries in some big tables so that when you search for “foobar”, it can tell you every post that contains that work. That’s a fine design for a small board with a tiny amount of traffic. But as your board grows, even legitimately, that table can become hundreds of thousands of rows long (or more!) and inserts and selects can become extremely slow.

It’s ten times worse when the only thing putting content into is spammers who are just flooding it with huge wordlists multiple times per second. Now, all of a sudden, you’ve got this single board showing up in your processlist five times, with each entry running for 30, 40, 50 seconds. One of those boards can cause some extra load on a server.

When you’ve got ten or twenty, it can bring the server to a halt. Literally. I popped onto a server where the load was near 10. I turned off 40 phpBB boards getting spammed. The load dropped to less than 1 and stayed there.

After some quick thinking, we identified a bunch of boards that were getting spammed and turned them off. One of our engineers built a brilliant little monitoring script that can identify phpBB boards in the processlist and shut them off if they show up at a high enough frequency with those awful queries (you know then when you see them, believe me). All told, we’ve turned off maybe 12k boards in the past 2 weeks, and haven’t heard a single complaint.

Why? Because these boards were setup by users who then forgot about them. And there they sat, for months or years, collecting spam, draining resources. Basic negligence on the part of users caused a huge server load, which then caused those same customers to call in and complain.

It feels like we’ve got this mostly under control, except for the user side. We need to figure out a way to get people to realize that the things they install on their site can be exploited and lead to security issues (on their site), performance issues (for everyone), and can suck up the resources they pay for.

But yeah, it sucks when you work about an 80 hour week because people forget about their phpBB install, and the folks who wrote phpBB decided that they’d build the most stupidly designed search setup of all time.

So, when on April 8th my Twitter looked like this:

now you know why.

I figured I’d log what I did when I upgraded my blog to WordPress 2.5.

First, I disabled some plugins I figured I wouldn’t necessarily need post-upgrade. The two I disabled were Kramer, which grabs Technorati links back to the blog (newly built into the WordPress dashboard) and SpotMilk, a customized dashboard (which I wasn’t even sure would work).

Then I upgraded.

So far, so good.

Poking around the settings, I decided to turn on the global Gravatar usage, rather than using the Gravatar plugin. That’s a great idea, except my theme doesn’t come with Gravatar support, so I’ll need to use the built in functions.

Then my MacBook crashed for the second time today (I think it’s Twitterrific, but we’ll see). Awesomely, MarsEdit earned its keep by having autosaved my work. So back to it.

After poking around, I got the built-in functionality to work, but since it returns an entire image tag, and not just the URL to the avatar image, it’s actually less useful to me than the plugin is. I turned the plugin back on. Good enough.

Next, I noticed the Mowser plugin had a new version. Perfect chance to try the new built-in plugin updating. Clicked the link and that was pretty much it — the plugin was up-to-date. Nifty. You can see the Mowser-fied version of my site here. Not perfect, but pretty good work from a one … two person company.

Took this as the opportunity to clean up my plugins page. Gone are the aforementioned Kramer and SpotMilk, along with the Hello Dolly and WP-flv plugin I’d installed a while ago.

Now, I wanted to turn some of my hard-coded plugin links into widget usage, to make switching themes easier. I started adding widgets to my left sidebar, expecting that I’d need to go disable them in the code. Nope! Nice, it must use a different bit of sidebar code when you use widgets. Very cool. This allows me to dump a couple more plugins (MyNetflix and a Last.fm one).

Also, turn off WP-Cache when you’re testing, or you’ll be annoyed out of your mind.

One missing widget: I was previously using the Google Shared Items widget, but now I’ll just use the RSS feed for it. Let’s see how that looks … ugly. But, good enough for now. Maybe there’s a WordPress widget for it. Wow, I’m digging the widgets. They make my life a whole lot easier. I should have tried this a long time ago. Even added a little About Me text widget.

Turning WP-Cache back on.

Finally, testing to see if MarsEdit can still post … huzzah! Success. And with that, I’m done.

I’ve been meaning for a while to sort of document how I get stuff done at work. It was just over a year ago that I bought my MacBook Pro. Within a week or so, I started using it at work. Probably within the first month, I’d completely moved to my MacBook as my sole work machine. After a year, and particularly since the upgrade to Leopard, I’ve kind of worked out how I get stuff done.

Let’s start with my environment.

The front wall of my office with pictures taken by a former co-worker. And, of course, the famous “Dwight” flasher flyer from “The Office”

The shelf behind me containing random stuff I’ve gotten from eating kid’s meals and ice cream sundaes. And some Yankee Swap gifts. Oh, and I have some windows. That makes my life nicer.

My white board and busted ass bookshelf. And my cool VT light switch cover from Matt, and some random stuff I’ve collected and hung up.

The view of where I sit. I used to use that big ass monitor to do a dual-monitor display, but since I’ve been moving around so much each day, now it’s just there to keep people from having a good look at me.

Finally, the MacBook Pro, my Motorola Q, my 30GB 5G IPod, my noise canceling head phones, and my phone that I don’t ever answer or use. And yeah, that’s Win2k running in Parallels. More on that in a bit.

So that’s where I do my work.

My Mac is setup in a very particular way. The upgrade to Leopard with Spaces has made my life considerably easier. It’s probably easiest to roll through how my Spaces are setup.

Space 1
This is where I use my browser, which is currently Firefox 3 Beta 3, and sometimes Safari 3.

Also running on this space is my “chat” clients. We use Jabber at work, which works nicely with Adium. I’ve also got Twitterrific running on this space.

Space 2
Here’s where my Terminal lives, which is just the default Leopard terminal. Tabbed Terminals make me happy, particularly once I made the default tab switching hot keys to be Command+Left and Command+Right.

It’s all command line and vim and mysql. Good times.

Space 3
Space 3 is where iCal and Mail live. Mail is just downloading my mail from Gmail. iCal is doing some cool stuff. I have most of my life in Google Calendar. iCal subscribes to my calendar feeds from GCal (including my work Outlook calendar–more on that in a second).

With all of my stuff in iCal, I then use the Missing Sync for Windows Mobile to sync my calendars to the previously mentioned Motorola Q (which also connects to my work Exchange server, so it’s almost as a good as a Blackberry).

Space 4
It’s the Windows space! I’ve got Win2k (don’t ask, I had a license lying around) running in Parallels, in Full Screen mode. Parallels runs pretty much just so I can run Outlook (for work email and calendaring) and so I can occasionally test stuff in IE6.

My Outlook runs a plugin called SyncMyCal to sync my Outlook calendar off to Google Calendar (which then gets sync’d down to iCal, as previously described).

Other software that occasionally comes in handy:

• NeoOffice (though it’s slow and bulky and I’d switch if there was a viable alternative)
• iTunes (obvs)
• MarsEdit (for doing this sort of stuff)

That’s how I get my work done. Anything else I should be using?

While I sit here and wait for myisamchk to finish and tell me that the table that various folks have spent the better part of the day trying to restore is either healthy or once again dead (how’s that for a run-on sentence), I wanted to dump out some of the things we’ve done to try to make our MySQL backend scale. It’s not been pretty, but given that it’s strung together with some Perl, some MySQL, and a bunch of paper clips, I think the folks around me have proven themselves brilliant (I just sit around and pretend to know what’s going on).

Oh, and this is all without a cluster. That’s probably the next step. And “Oh: part 2,” myisamchk finished checking 28 million rows. All is good. I’ve copied the data over to the main server and brought it up and everything is happy. Back to MySQL scalability …

First issue: We’ve got too much data

This one was easy to solve. We got rid of it. Sort of. We started archiving off data that we no longer needed chronologically. Old support incidents, logging, anything that had a timestamp that we aren’t looking at gets sliced off into an archive so that the main tables can be as tidy and fast as we can get them. Which for us is like 8GB of data and not fast at all. But it’s better than 12GB of data.

Second Issue: We’ve got too many connections

When you’re small(ish), it makes sense to throw a bunch of dbs on the same server. As you grow, those connections start to swamp MySQL. MySQL starts to get all panicked, and it doesn’t know how to handle all of the people asking for data, so it starts to get sloppy about closing old handles. Then it’s basically like thermal runaway in a transistor. The server can’t close old connections, new ones open up, adding more overhead, and all of a sudden your nice server has 5000 open connections and is hosed. Again, this was a pretty easy one to fix. Bring up a new box, move some databases it to it, and hope that you’ve built your code layer to make that swap pretty easy (ours was). Presto. Now both of your servers are happy.

Third Issue: We lock up the damn tables all the time

We’ve got a lot of customers who are constantly accessing their sites. We’ve got nearly 1000 support agents across the globe who are using our tools to look at customer configuration to make sure there’s no issues. This puts a whole chunk of load and repetitive queries on the database. That’s easily handled.

Except when you add in a bunch of data updates. Agents, customers, new signups adding and editing data in the database. All of a sudden those hundred pending SELECT statements are stuck because one big select locked the data when an UPDATE came in. Now you’ve got a bunch of web users who think your stuff is slow and/or broken. We’ve tried to attack this in a few ways:

1. Fix your queries — We watch our slow queries and try to make them faster. We look at our most often called queries and try to make them faster. Sounds simple, but bad queries are the biggest cause of problems.
2. Add indexes — This goes with “fixing” queries. Add indexes and make sure you’re queries use them.
3. Perform less queries — Can you cache your data? Can you make less queries and do more in your language of choice? Can you make your users smarter (maybe without them knowing) about when they need to request data? Do it. The less queries you have, the more likely you won’t lock things up.
4. Split your reads and your writes — If you can split your reads and writes at the code layer, then you can shuttle reads off to one (or more boxes) and writes off to the primary box, and you should lock up a lot less. We accomplished this by having a couple of boxes replicate the main database, and having one of our smart engineers subclass Perl::DBI to look for SELECT statements and swap the database handle over to the read replica. It helps more than you might thing (but it’s not a silver bullet).

Most of this sounds like common sense. It is. But it still matters. We’re trying to do a lot with a little, and every ounce of performance you can squeeze out matters, when your users are super demanding and will use any slowness as an excuse.

There are some other things we should and probably will try:

• Denormalization to bring data back together and cut down on costly joins
• Sharding to split our data up into smaller chunks and this cut down on long table scans and huge indexes
• A real MySQL cluster to optimize reads and writes and spread traffic out to many nodes

I wish I knew more. I’m still barely up the curve compared to some of the engineers and admins I work with. Thankfully, they’ve been able to keep our many million row tables (and many GB data and index files) humming along with few interruptions.

Preface: These comments are mine and not those of my employer or anyone else. Ok, maybe they also represent the voices in my head.

Advertising on the Web

There’s lots of advertising on the web. The biggest web company in the world (Google, ever heard of them?) generates pretty much all of their revenue from text (and now some image ads) based on the context of your web searches, email, or web page content. Big media sites like ESPN or ABC or NBC generate some revenue and awareness through the old late 90s staple of the banner ad. Blogs, podcasts, and video sites get in on the action with pre/post-roll ads, typical interstitials, and sponsors. Tons and tons of ways for sites to generate some income, but they’re all pretty much based on getting a large number of eyeballs.

With the recent growth of blogs, forums, and just the general smaller sites run by individuals rather than corporations, folks have wanted to cash in on some of that free internet money. But banner ads and AdSense cash really don’t work too well unless you gets loads of traffic. Now, granted, there’s tons of ways to do that (which is why a bunch of the junk that fills up sites like Digg and Reddit these days are obvious linkbait bullshit attempts to generate lots of traffic), but the internet in general–through better algorithms and crowdsourcing and such–has gotten pretty good at weeding those things out. Besides, only a few linkbaiting attempts can work in a given time period, so even this is not a surefire way to get yourself that sack with a dollar sign on it.

Affiliate Marketing: Good

This gave way to a new niche: affiliate marketing. In all honesty, this started a while back with Amazon. And, to this day, Amazon’s method isn’t really bullshit. Amazon’s affiliate system and basically a way for people who review stuff or talk about different things they want to link to those items on Amazon and get a kickback if someone purchases through their link. Everybody wins in this situation; the buyer gets an object they wanted, the site owner gets a few bucks back for setting up Amazon and the buyer, and Amazon gets to sell an object.

Plus Amazon gets some search engine love from having lots of sites link to them.

I’ve used the Amazon affiliate system when doing my not-very-often-update podcast. I’ve never made a dime, but that’s because I don’t do it very often and I’m not exactly linking to highly sought after stuff. But disclosure is important, because it’s the lack of disclosure is a big reason that affiliate marketing is currently the <insert your horrific disease here> of the interweb.

Affiliate Marketing: Bad

Somewhere along the way, the affiliate stuff got bastardized. It was, of course, inevitable. We live in a world of pyramid and get rich quick schemes broadcast in half-hour increments on late night TV. But the internet’s version is far more nefarious. The BS seemed to start in earnest with BzzAgent, a Boston-based marketing company that paid “agents” to go around talking up products–products that the agency had been paid to promote. There was no disclosure; “Hey, I’ve never actually tried this product! I’m getting PAID to tell you that I think it’s awesome!” Of course, BzzAgent took some heat for their arguably deceptive marketing. It was not intentionally deceptive, but they were implicitly offering people incentives to be deceptive. After some bad press and some backlash, BzzAgent claims to be all about the disclosure.

Affiliate Marketing: Worse

Similarly, a company called PayPerPost sprung up. Here’s a group that will pay you to write about a product, service, or other site, right on your own website! You write up a few paragraphs, throw in a few links, and you make some money. The sponsoring company gets some search engine juice and some good word of mouth. PayPerPost gets some money for bringing the two parties together.

Not so different from the Amazon model, right? Sure, except that there was no required disclosure that the post was, basically, just a paid advertisement. Posts from PayPerPost folks weren’t required to be tagged as advertising, the way that those fake magazine articles are. The writer never even needed to try or use the product they were writing about. It was obvious to everyone what was going on: blatant link buying in an attempt to game the search engines.

(For a more complete story on PayPerPost, try TechCrunch.)

So what’s the difference between this and the Amazon model? The end-user. The buyer. They’re getting hosed in that they’re just the commodity being traded in the middle. Taking someone to Amazon, where they’re then exposed to any number of other reviews for the product in question (which, by the way, is almost always a consumer product that’s a tangible good) is incredibly different than linking them to a web hosting company (more on that later) or some other digital good that is not quite as easily identified as something a user does or doesn’t want.

Particularly when all of the reviews for said product are paid for, and thus biased, by the aforementioned affiliate system.

So BzzAgent and PayPerPost started paying people to write about products and services, without a requirement of disclosure, and in many cases, without actually even trying out the product they were promoting. Sure, they were just trading on their online identity–burn people enough and you’re opinions are worthless. That is, of course, unless you can create endless domains and identities. The two companies were rightfully shat upon by the honest folks on the web. Both have started to talk about honest disclosure and transparency in attempt to stay relevant and to ensure their clients don’t run away for fear of being painted with the same dishonest brush.

It hasn’t worked for PayPerPost, whose business was rightfully crippled by Google when Google basically dropped the rank of any site found to be working with PayPerPost. If PayPerPost’s business is as honest as they claim it is, this wouldn’t have mattered. The paying companies would still be lining up to get reviews and links. They’re not. They wanted search juice. And that’s not for sale, well, not through PayPerPost, at least.

Affiliate Marketing: The Drizzling Shits

Which brings me to my biggest pet peeve, and the one that hits closest to home: bullshit web hosting review sites. There’s tons of them. They claim to review web hosts. They don’t. They rank sites based on who pays them the most money per hosting sign up. It’s, quite frankly, a pox on the hosting industry. Each web host offers the affiliate a little more money. In return, the affiliate gives them good links for SEO, some traffic and new sign ups, and a couple of web STDs.

From some of our internal research, somewhere around 20% of all sign ups that come through affiliates are fraudulent. Most of them have a life span significantly shorter than a typical sign up. Many of the sign ups that make it through the front end fraud checks are still BS accounts. They sign up, collect the affiliate fee, and then cancel. With most web hosts, if you did that 10 times a day, you’d make in the neighborhood of $100k a year.

I’m not kidding.

Why is this so bad? Again, it comes down to disclosure. None of these sites reveal they’re doing this for pay. Most of them layer some arbitrary, made up review score on top of their listings, depending on which host is paying the most that month. The affiliate doesn’t care that it’s slimy–they’re getting paid. The web host doesn’t care that it’s slimy–they’re getting new “real” hosting accounts. Who cares? The actual honest person who did hit Google or their search engine of choice to look for a web host to open a blog or a place to host their pictures of their grandkids. They find a review site, sign up with the top rated host (”oh my, this host is rated the top on ten different sites!”), and then find out it’s a completely crappy host. The poor grandma doesn’t realize that ten different review sites were all run by the same person/group/company. She didn’t realize that the top host was paying these affiliates so much because their service is so bad they’re hemorrhaging customers.

I’ll admit, my company pays affiliates. Slimy ones, at that. We’re not hemorrhaging customers. We’ve actually stepped up our game, I think, and have started to deliver a better hosting experience for most of our customers. But growing organically by word of mouth isn’t good enough for us, so we put on the full body web condom and deal with the underbelly of the internet.

It’s disgusting and immoral and we shouldn’t do it. Many of us have made that case. But, unfortunately, the dollars trump us. So we build in workarounds and special rules to pay off certain affiliates to make sure they get the conversions they want so they’ll keep sending us traffic. And keep linking to us.

Affiliate marketing isn’t inherently bad. But, as with anything, when you mix it with the internet, it ends up being more bad than good. It’s the drizzling shits of the internet.

Soon enough, Google will step up and kill this trend. And it’ll be a great day when we can focus on stuff that matters and not spend thousands of man-hours building search algorithms to weed out fake sites, building fraud detection to weed out the fake sign ups, and trying to convince ourselves that just because other folks are doing it, we need to do it to keep up.

Yuck.

Save Us Obi-wan

Dear Matt Cutts,

Can Google please fix the fake review sites? It would be awesome.

Thanks.

Your pal,
The Interweb

Affiliate marketing is everywhere now. Google it. You’ll find hundreds of blogs devoted to how to get a spammy, content-less site ranked high in the search results, get people to click your links to generate conversions, and how to basically make money being dishonest. Granted, all marketing is somewhat dishonest–promote the good stuff, hide the bad stuff. But when it’s a first party doing it, you know to take what they say with a grain of salt (which is why good companies are transparent and talk about their occasional foibles … it makes the marketing spin look less spinny). When a supposed neutral third party is hiding the fact that they’re making money off of their “review,” it’s not easy to discern that. It’s ugly and stupid and dishonest. And it makes people loads of money.

Again, it’s why affiliate marketing is the drizzling shits.

Examples

Just in case you’re wondering, here’s what a bullshit review site looks like. I shouldn’t claim this to be authoritative. I don’t know with 100% certainty that these are fake review sites. But they fit the mold. They cloak their affiliate links, bring you over to the web host with an affiliate cookie, and have surprisingly similar reviews. I won’t link to them, but you can paste them into your address bar.

http://www.best-webhosting2007.com/
http://www.web-hosting-review.toptenreviews.com/
http://www.web-hosting-reviews.org/
http://www.web-hosting-top.com/
http://www.webhostingtoplist.com/
http://www.webhostingfever.com/
http://www.websitehostingreviews.com/
http://www.100best-free-web-space.com/

The company I work for has been growing. Rapidly. Our business model (as I think I’ve mentioned before) is that we acquire other hosting companies and merge their customers into our pretty scalable, manageable platform. It’s generally a ton of work, replete with headaches, stress, and lots of long hours.

But, at the end of the process, we’ve got a bigger company, more customers, new tools and applications for our entire base, and a few months to stabilize and work on new projects. It’s a hard cycle, but one that works well, and one that has worked exceedingly well because we’ve kept our team small and focused. Pretty much everyone knows what’s going on, is clued in on the ins-and-outs of the platform and any new changes, and understand the implication of every decision. It was an ideal situation. When you had to cut features to hit a deadline, drop support for something because the work expended dwarfed the number of customers who used it, or changed the way something worked–everyone was on board. When work ramped up, people who hadn’t previously been involved could pitch in and help out without making this later (i.e. defeating that rule established in The Mythical Man Month). A small team, busting ass, getting really hard tasks done just under the wire.

Our new acquisition makes our company significantly larger. The process of moving customers is also significantly larger. We’ve added a ton of staff across the entire organization to theoretically help make things work better/more smoothly/with ice cream and puppy dogs.

It just doesn’t work that way. You lose the closeness of the team. The ability to communicate quickly and have everyone on the the same page. You add more people who need to be trained on the existing platform before they can even start working on new stuff.

Sadly, you add people who just don’t really care at all about pitching in.

With a small team, people can’t really hide. If you’re not pulling your weight, it’s obvious, and your peers kinda take care of it. As you grow, people figure out how to duck out, stay invisible, do the bare minimum to get by without being noticed.

This is going to sound conceited, condescending, and douchebaggish, but I think the folks who get in the way as you grow generally fall into four categories.

1. New people who are just too new to help out
   Obviously, you can’t blame these folks. They’re coming to help out, but they just need to get up the learning curve before they can be counted on to be effective without being a drain on the team (i.e. the Fred Brooks’ rule that adding people to a late project just makes it later). If you’re lucky and you hire well, these guys will come through the first month or so of their employment sucking up as much info as they can, ready to help out within 4-6 weeks. But, with most new hires, it’s a bit steeper curve, and you’ve got some folks who are anxious to pitch in that you need to keep diverted on other stuff.
2. “That’s not my job”
   As you grow, you tend to hire people into specialized roles. When you’re small, everyone’s a generalist. Everyone can do everything, and will pitch in wherever they can. Eventually, you cross the threshold and start to hire people who are specialists and have no desire (or ability) to generalize. So when you need to spread the load for a big project (or lots of little projects) across your newly enlarged workforce, you all of a sudden encounter a new set of responses:

   “That’s not really what I do.”
   “I’m not really interested in working on that.”
   “I’ll try, but I don’t think I really understand what you mean.”
   “That’s not my job.”

   Obviously, that sucks. It’s crunch time, you’re reaching out to people, and they can’t get over their job title enough to pitch in. You just have to hope you’re smart enough not to hire many people like this. There’s a slight alternative to this person ….

3. I don’t have enough work to do so I make up my own
   “What?” you say, “How could someone not have enough work to do?” Well, sometimes people are specialized. People who’ve been hired and pigeonholed because they’re really not that good at their job, or because they’re a pain in the ass to work with, or <insert your reason here>. Often, these people will (rightfully, and properly) want to do something. The problem is they’ll go hunting creating a problem so they can make up their own solution. They’ll start to churn up trouble (”I don’t really like the way this works. I think it should do this.”) or start working on changes to things.Normally, this isn’t too bad. New ideas are generally welcome. In fact, a new perspective will almost always bring with it some nice nugget that allows you to make your stuff better.

   The problem is that it happens in the middle of crunch time when your focus needs to be on the project at hand. It’s further complicated because the people who start looking for problems in need of solutions, often try to solve problems that they don’t quite understand. They may not have been involved in the initial project(s) and thus don’t get that the problem they’re “fixing” is unfixable for technology, platform, manageability, or any number of reasons.

   You end up with people who are further isolated thinking their ideas aren’t valued and your focus is diverted from the necessary work.
   
   

4. It’s Miller Time!
   When you get bigger, people lose the mentality of sticking around and pitching in until the work is done. Granted, everyone needs to get out of the office. And there’s nothing wrong with working a 9 to 5. But on big projects, particularly ones that affects tens or hundreds of thousands of customers, sometimes you need to stick it out.There are those folks who look at the fact that there are people leaving, so even though there’s some project work left, they duck out. Or they say “hey, there’s a few people left, they can handle it.”

   The result is a bunch of people who are pissed because they’re staying and doing the lion’s share of the work, and another chunk of people who are alienated because they don’t think people should expect them to have to stay and work late.

In all of these cases, it’s pretty easy to see both perspectives. And my intention isn’t to be an ass and call out the people in these groups. It’s simply a set of observations from a project manager/pseudo-engineer who’s been working at a company that’s grown from 40 employees in one office, to 100 or so spread across the country.

So, it all leads back to the question at hand:

Would you rather be overworked or homicidal?

Would you rather work at a small company, busting your ass, feeling stressed and overworked, but knowing everyone is putting in 100%? Or would you rather work at a larger company, with a ton of people to do the work, but ready to throttle those folks who aren’t pulling their weight?

I’m in the former camp. It could be an age or life station thing (though, it doesn’t look like it, in my limited experience), but I’d rather just be working somewhere I can bust ass and know that everyone else is too. Small teams or small companies or startups.

I think that my company will figure out how to make it work. It’s just going to be an adjustment. There’s going to be some growing pains (there already have been, as we’ve lost a couple of good people). As long as we don’t lose any more good people, we should come out of this and understand (hopefully) that our current strategy doesn’t work.

We’ve been a textbook case of forgetting the rules of The Mythical Man Month. Maybe that’d be a good gift for some folks in our company.

We had a database issue the other day. We were losing records in our DNS database. It turned out the table was corrupt and inserts were failing.

We used MySQL’s REPLACE INTO to update the table. So, in essence, this is what was happening:

try to insert domain
oh, I can’t insert? that means it’s already there
delete the existing record
try to insert again

On a happy table, that works just fine. The INSERT fails initially because the table is corrupt, but REPLACE INTO thinks it is because the record exists. So it DELETEs it. Then it tries to futilely INSERT the replacement record, which obviously fails.

Now we’re down one record. Oops.

Thankfully, it was easy for us to fix the table and replay the changes and get everything back to normal. Yay. Filing this one away in my brain in case I run into it again.

My company is at the beginning of a what will end up being a fairly long, exhaustive migration process. Probably on the order of 12-16 months, migrating web sites from a set of servers on one side of the country to a set of servers on the other side. It’s not your typical forklift migration (where you actually move the servers and plug them in at their new home); instead, it’s literally moving files, mail, DNS, etc. to a new platform.

It’s pretty daunting, pretty complicated, and can occasionally be pretty cool.

On the flip side, it’s now 2:55AM Eastern in Boston (where I started my day), but I’m in Phoenix where it’s actually only 11:55PM. That’s a sign that maybe things didn’t go quite as smoothly as one would have hoped.

The step we’re on is a step where we take over DNS for folks. It’s always somewhat difficult, because we’ll get a big list of domains and have to figure out whose record (our nameserver’s or the other nameserver’s) is the “real” record. It’s not generally too tough to figure it out (you can judge by the SOA of the records) and the number of domains is usually short of 100k, so as long as you’re accurate to within 1-2%, it’s not too bad. That’s 1000 guys who might break, which is pretty easy to handle with a good support team and some quick script fixes.

Let me take a step back. The process is actually that our nameservers need to become the authoritative nameservers for the domains we’re moving. This allows us to later change their DNS to point to their new home, and it all kinda works. We have to get the domains, merge them into our nameservers, become authoritative, and then fix what breaks.

We did that yesterday. Except it wasn’t 100k domains. It was 1.2 million. And the domains weren’t coming from a single, well-maintained nameserver. They domains came from three, somewhat munged together nameservers. There were internal conflicts, conflicts with our servers, missing zones. A host of issues. We thought we’d worked most of them out and gotten the problems down to, at most, 4-5k domains. That’s a lot, but in reality, it’s less than 0.5% of the total domains.

“Pretty good,” you say.

“Not quite,” I say.

For you see, there weren’t just three nameservers. There were five. So there’s a couple thousand domains we missed. And we also missed some of the conflicts (either by omission or by grabbing the wrong data). In the end, it was closer to 12k domains that were wonky.

That’s still only 1%. Damn good, given all of the variables.

Except 12000 broken domains leads to a whole lot of phone calls and emails. And some angry customers. And some tired folks staying up to fix things that they weren’t responsible for breaking. And one tired folk–me–staying up because he feels guilty for only being 99% good enough.

DNS is a fickle beast. Thankfully, it’s pretty quickly fixable. Once we’d identified some global problems, we could fix them rapidly and put big chunks of the broken domains back in working order.

I often argue with people who think the “Chinese Market” is a valid business plan. You know, the folks who say “hey, if we can just get our product in front of 100 million people, and get 1% of those people to buy, we’ll be rich!” Except, of course, it doesn’t really work that way. It’s hard to get a product in front of that many people who would be interested in buying, and it’s hard to get 1% of any audience to buy anything.

Well, not in business plans, at least. It does work that way in technical issues. If you’ve got a huge enough base of users, the smallest mistakes can have a big impact on your company and team. In these cases, sometimes being 99% accurate isn’t good enough.

Here’s a graphical representation:

You see, with 100k domains, you never quite reach screwed. It’s manageable.

With 1.2 million domains, you’re pretty much totally screwed.