Though this made me laugh ….

Well, it would have made people laugh if I had the ability to upload a file … so instead, I’ll just paste the text in:

12:35:39 up 8 min, 3 users, load average: 413.01, 255.54, 113.59

Good times.

Oh … it’s getting worse:

[flower]$ uptime
12:46:20 up 18 min, 3 users, load average: 742.20, 567.02, 336.31

That’s when I realized how much Twitter had taken over.

I hadn’t logged into my web interface in a bit, so I had a WordPress update, 4 plugin updates, and a bunch of pending spam. I cleared that out and then started crafting a post in MarsEdit. As I went to post those fateful words, MarsEdit choked on an XML-RPC error complaining about bad content.

I quickly scanned through recent post, looked for bad characters. I used the Googles to try to find an explanation. Finally, I noticed that each page of my blog was spitting out a PHP warning (because, you know, PHP is pretty dumb, and I’m too lazy to have turned off warnings) that it couldn’t download my latest tweets from Twitter.

One quick click of “Disable” and MarsEdit sprung back to life.

Twitter was breaking my site.

At which point, it dawned on me, that at this point, Twitter being down is like having part of the internet’s routing being down. It’s tied into so many systems, and so much traffic/content flows through it, that when it goes kablooey, all that content has to route elsewhere. Which then floods those systems, and they start to struggle and burst at the seams a bit, and then folks overflow into another system, and so on.

Until, the end result, of course, is that Twitter was breaking my site.

Here’s a graphical representation:

It’s fixed now.

My site, that is, not Twitter, which is still down.

If you do any web development, this tool is awesome.

But, let’s start with the basics …

Typical Web Host
A typical web host has a “pizza box” architecture. They pile a bunch of servers in a rack in a data center. Each server is running some MySQL, some Apache (or ISS), some email app (Exim or Qmail), and some FTP. Each server is usually running another instance of Apache to run the customer control panel. The box usually runs off it’s own local storage. Thus, each box is an individual entity running it’s own versions of applications, with it’s own individual issues. Depending on which box you’re on, you might have MySQL slowness, or Apache slowness, or disk slowness, all depending on what your neighbors are doing. Rolling out upgrades becomes a bit more arduous as you’ve got to upgrade each box. This is decidedly old-school, but very common and works fairly well. This architecture only starts to be a problem as you grow; each time you add X customers (where X is the number each box can support), you have to add a new box. Each time you add a new box, you need more power, space, backup storage, etc. That gets hard to scale.

So, then there was the idea of not using local box storage, but instead using networked storage. Now you can just add more disks without having to add entirely new boxes, and backups become less of an issue. This means you can grow a little bit more without having to add more boxes.

Atypical Web Host
An atypical web host (like us and some of our competitors) might do things a bit different. For instance, rather than having a single LAMP (Linux-Apache-MySQL-PHP/Perl/Python/(P)Ruby) stack per set of customers (on a box), you build a pool of boxes that can all serve up services for a customer. This means that:

• You’ve got some redundancy
• You don’t have to scale all of your services as you add new customers

In this model, more like a typical web service than a web host, you’ve got pools of servers that perform certain tasks, and customer data is kind of abstracted away to all just live on big storage arrays. Things scale much easier and more cost effectively.

But ….

What does this have to do with Twitter?

See, when you’re running centralized services, you end up with a lot of data sitting in your database, and it’s getting accessed by Y thousands (tens of, hundreds of) users. The “pizza box” model doesn’t have this issue since the data is distributed across thousands of servers. But that’s not feasible in something like Twitter. You don’t only want to be able to see tweets from people who are on the same server as you. You want to see tweets from everybody. That’s the power of Twitter.

Having lots of data in a single database instance solves that problem. But it introduces a whole new set of issues: database locking, slave synchronization, disk performance.

Let’s take an educated guess at what Twitter’s database schema might be like:

Table: Users
UserID     int(14) autoincrement
Username   varchar(50)

Table: Tweets
TweetID	   int(14) autoincrement
UserID     int(14)
Tweet      varchar(140)
DateStamp  datetime

Table: Followers
UserID     int(14)
FollowerID int(14)

Table: UserTweets
UserID     int(14)
TweetID    int(14)

In a nutshell, we’ve got a table that keeps track of our users. We’ve got a table of tweets, which maps back to give us a user based off of the UserID. Right there, you get a pretty easy query to get all of the tweets for a user:

SELECT t.Tweet FROM Tweets t INNER JOIN Users u USING (UserID);

When Twitter was just a baby, that query’s pretty darn fast. It’ll give you all of the Tweets written by any user in particular. Reads will be fast, since UserID will be indexed and unique. Writes will be fast, since we’re not writing a ton of data and updating the indexes should be pretty quick (since Twitter is still just a baby).

Now, we add the functionality of “following” another Twitter user (good thing we thought ahead and added that to our schema!) Now a user can follow another user, which just simply sticks a row in the table with the id of the user and the id of who they are following.

Keeping track of all the tweets flowing into the user from folks they’re following is easy, since we planned ahead. We made a table that lets us map a bunch of tweets to a user. Every time we add a tweet, we stick a row in that says “this user has this tweet”–it doesn’t matter if the tweet was by that user, or someone they were following, they all go in that mapping table.

Again, when Twitter was a baby, this was all pretty easy and quick:

SELECT t.Tweet FROM Tweets t INNER JOIN UserTweets u USING (TweetID) where UserID = ?;

We’re going to get back all the tweets assigned to that user, whether written by the user or someone they are following. Simple and fast, since again, we’re hitting some pretty easily indexed and unique fields.

Of course, I’ve made one assumption here. I’m assuming that, when a user starts following someone, that their tweets start getting associated to my user (the UserTweets table). I’m guessing this is the case since when you start following someone all of their tweets don’t magically show up in your history (conversely, when you stop following someone, they don’t miraculously disappear — I don’t think so, at least). Either Twitter sticks stuff in the mapping table (my guess), or they go look up the date you started following each user, and they build that index on the fly.

In other words, the alternative version is that Twitter, when you view a page that displays your tweets along with those of folks you follow, would have to go lookup the date when you started following someone, then find all of their tweets after that date, bring them together, and put them in order.

Mapping table seems a whole lot more likely. And much faster. Which is kind of how Twitter got big.

Anyway, since we’ve got this nice mapping table, we need to know how to fill it up. If I’m following Robert Scoble (like the rest of Western Civilization), every time he writes something, it needs to make it to UserTweets associated with me.

Again, this is pretty easy, if you’ve got a handy loop. First, get all the followers:

SELECT FollowerID from Followers where UserID = SCOBLES_ID;

Then, loop through all those folks and post it:

INSERT INTO UserTweets (UserID, TweetID) values (FOLLOWER_ID, SCOBLES_TWEET_ID);

That loop will run as many times as it takes to update Scoble’s followers. None of these queries are complex. They’re all easily indexible. They should all be pretty fast. When Twitter is still in baby-state.

But now Twitter is growing. It’s a toddler. It’s got many many users and some of them have loads of followers. Things are starting to slow down. Twitter’s parents look at it and say “Well, one obvious reason you’re starting to slow down is that the database is starting to lock. See, now that we’ve got some data, lookups are taking a little bit longer, and inserts are taking a little bit longer, and when they happen at the same time, we don’t want data to get out of sync, so the database locks up. When it locks up rapidly enough and often enough, we run out of threads and our database likes to go down. We’re going to teach you to read and write at the same time.”

And that’s what they do.

We currently use one database for writes with multiple slaves for read queries. As many know, replication of MySQL is no easy task, so we’ve brought in MySQL experts to help us with that immediately. (blog.twitter.com)

Now we’ve got a master database, where all of our writes (inserts) go, and a couple of replicas where all of our reads (selects) go. Perfect. Things are fast again. Users are happy. Twitter is moving along.

Twitter grows into the tween stage. And it’s not pretty. Databases are constantly crashing. Things are slow. What the hell? Didn’t we just fix this?

Well, no. We just hid the problem. Replication isn’t a pretty solution. MySQL replication is flawed. It’s not instantaneous. It can fall behind. Further, it breaks. A lot. And when it breaks, we lose half of our read capacity, which then overloads the other server, and everything goes down. Resynchronizing can be painful. Adding more replicas helps, to a point (since each replica adds a little bit of overhead to the master).

Big users (those with lots of followers) cause more load, since now we’ve got to do a big lookup to get thousands of rows (followers) and then do thousands of inserts every time one of those users posts. That’s not good for our database.

NOTE: The Twitter folks claim they don’t copy the tweet around for each follower. I’m sure they don’t. But they don’t say anything about copying the ID around (which, as I’ve stated, makes a good amount of sense if you were architecting Twitter 18 months ago).

Also, it’s what Dare Obasanjo posits, building on what Israel says on the Assetbar blog, and they’re both way smarter than me.

And you can’t really add a second master database. So writes are going to be as fast as our server can handle them.

This is where we are today (or at least recently). There’s enough users on Twitter that updates are probably starting to slow since the tables and indexes are so large that there’s just not much MySQL can do. Reads are slow because replication is fragile.

What do we do?
Without completely re-architecting things, what can we do? There’s a couple of things, right off of my novice brain:

• Cache, Cache, Cache
• Make the database smaller

Cache, Cache, Cache
The less we have to go to the database, the better things perform, and the better they scale. There are lots of things we can cache, things which don’t update very often. For instance, the list of followers. If we stored the follower list in memory for each user (as it was loaded), we’d cut down on a query that gets run every time a user loads Twitter.

That’s *a lot* of queries.

It sounds like the folks at Twitter are already heading down this path.

A: We’ve mitigated much of this issue by using memcached, as many sites do, to minimize our reliance on a database. (blog.twitter.com)

Make the database smaller
This isn’t something it sounds like Twitter wants to try. Basically, this is sharding. You take all the users with a UserID < 100000 and put them in one database, all the users with a UserID > 100000 and < 200000 and put them in a second database, and so on. Then you have a master lookup that lets you know where to find the data you’re looking for (which you can cache!).

This makes your selects and inserts fast again, since the tables and indexes are smaller. But now you’ve got to manage the multiple shards, adjust them when they get too big, and deal with the added layer of abstraction.

It’d help, but it’s a big task.

If you’ve been reading my blog, you might be asking yourself:

Why Does This Sound Familiar?
Why does this sound familiar? Because our unknownish web host ran into similar issues. See my post Thoughts on MySQL Scalability From a Certified MySQL Moron from February.

All of the problems Twitter is facing seem to be directly related to building the service in a logical fashion, but not forseeing the problems that massive growth would have on the system (a single tweet spawning tens of thousands of database updates). Quite frankly, it’s completely reasonable. Frustrating, but reasonable. As Michael Kowalchik (founder of Grazr) states:

As a startup there’s only so much energy you have, and you must apportion your resources carefully. The truth is, we like to talk about scaling, but without steady growth and something people find compelling, all the scaling in the world won’t help you. (mathewingram.com

I’m curious about Twitter’s solutions both as a user and as an engineery web dork at a company hitting the same problems. We’re starting to use memcached, looking at using smarter non-table locking databases, and thinking about utilizing the file system more than a database.

Twitter has some technological mountains to climb to be able to scale to support the rate at which they are growing. I think that, sooner or later, they’ll have to bite the bullet and use database shards. They’re also likely going to have to build out memcached clusters large enough to allow them to cache nearly every thing about a user. That’s loads of data (gigs and gigs and gigs), but machines and memory are cheap for a company like Twitter, and the payoff will be worth it. I have no idea if Twitter’s growth is slowing during these performance issues, but throwing some funds behind a massive memcached cluster would be well spent now, and would surely be useful even after any sort of re-architecture.

Then, if you want, you can do some fun stuff to it. Like add some comments:

Tada!

Awesome. Takes about 2 seconds.

Our setup is a bit different than most. Since we don’t run a typical box-by-box web hosting architecture, we don’t simply have a thousand boxes with each one running Apache and MySQL. Instead, we have a really robust pooled architecture for everything except MySQL, which just isn’t something that’s very poolable. For MySQL, we’ve got some big boxes with a bunch of memory and some fast disks that handle our MySQL load. But, slowly over time, performance had degraded.

When you’d hop onto a box and look at the transactions per second or number of queries, nothing looked terribly out of the ordinary. Yet the load would be huge, and performance would be pretty bad. Our team brought up some new boxes, shuffled customers between them to even the load out, moved our backup processing onto the hot spare replicated boxes (to reduce even more load on the disks) and things were better.

But they weren’t better enough. (I know, awesome English, eh?)

We started just watching the processlist, looking for the culprit. And after about 5 minutes, it was obvious.

Motherfrakking phpBB spam.

phpBB is written in a really shitty way. Not the forum part, necessarily, which works when it’s not being exploited. But the search part is awful. For every word in every post (unless you’ve got a smart list of words to ignore), it throws entries in some big tables so that when you search for “foobar”, it can tell you every post that contains that work. That’s a fine design for a small board with a tiny amount of traffic. But as your board grows, even legitimately, that table can become hundreds of thousands of rows long (or more!) and inserts and selects can become extremely slow.

It’s ten times worse when the only thing putting content into is spammers who are just flooding it with huge wordlists multiple times per second. Now, all of a sudden, you’ve got this single board showing up in your processlist five times, with each entry running for 30, 40, 50 seconds. One of those boards can cause some extra load on a server.

When you’ve got ten or twenty, it can bring the server to a halt. Literally. I popped onto a server where the load was near 10. I turned off 40 phpBB boards getting spammed. The load dropped to less than 1 and stayed there.

After some quick thinking, we identified a bunch of boards that were getting spammed and turned them off. One of our engineers built a brilliant little monitoring script that can identify phpBB boards in the processlist and shut them off if they show up at a high enough frequency with those awful queries (you know then when you see them, believe me). All told, we’ve turned off maybe 12k boards in the past 2 weeks, and haven’t heard a single complaint.

Why? Because these boards were setup by users who then forgot about them. And there they sat, for months or years, collecting spam, draining resources. Basic negligence on the part of users caused a huge server load, which then caused those same customers to call in and complain.

It feels like we’ve got this mostly under control, except for the user side. We need to figure out a way to get people to realize that the things they install on their site can be exploited and lead to security issues (on their site), performance issues (for everyone), and can suck up the resources they pay for.

But yeah, it sucks when you work about an 80 hour week because people forget about their phpBB install, and the folks who wrote phpBB decided that they’d build the most stupidly designed search setup of all time.

So, when on April 8th my Twitter looked like this:

now you know why.

First, I disabled some plugins I figured I wouldn’t necessarily need post-upgrade. The two I disabled were Kramer, which grabs Technorati links back to the blog (newly built into the WordPress dashboard) and SpotMilk, a customized dashboard (which I wasn’t even sure would work).

Then I upgraded.

So far, so good.

Poking around the settings, I decided to turn on the global Gravatar usage, rather than using the Gravatar plugin. That’s a great idea, except my theme doesn’t come with Gravatar support, so I’ll need to use the built in functions.

Then my MacBook crashed for the second time today (I think it’s Twitterrific, but we’ll see). Awesomely, MarsEdit earned its keep by having autosaved my work. So back to it.

After poking around, I got the built-in functionality to work, but since it returns an entire image tag, and not just the URL to the avatar image, it’s actually less useful to me than the plugin is. I turned the plugin back on. Good enough.

Next, I noticed the Mowser plugin had a new version. Perfect chance to try the new built-in plugin updating. Clicked the link and that was pretty much it — the plugin was up-to-date. Nifty. You can see the Mowser-fied version of my site here. Not perfect, but pretty good work from a one … two person company.

Took this as the opportunity to clean up my plugins page. Gone are the aforementioned Kramer and SpotMilk, along with the Hello Dolly and WP-flv plugin I’d installed a while ago.

Now, I wanted to turn some of my hard-coded plugin links into widget usage, to make switching themes easier. I started adding widgets to my left sidebar, expecting that I’d need to go disable them in the code. Nope! Nice, it must use a different bit of sidebar code when you use widgets. Very cool. This allows me to dump a couple more plugins (MyNetflix and a Last.fm one).

Also, turn off WP-Cache when you’re testing, or you’ll be annoyed out of your mind.

One missing widget: I was previously using the Google Shared Items widget, but now I’ll just use the RSS feed for it. Let’s see how that looks … ugly. But, good enough for now. Maybe there’s a WordPress widget for it. Wow, I’m digging the widgets. They make my life a whole lot easier. I should have tried this a long time ago. Even added a little About Me text widget.

Turning WP-Cache back on.

Finally, testing to see if MarsEdit can still post … huzzah! Success. And with that, I’m done.

Let’s start with my environment.

The front wall of my office with pictures taken by a former co-worker. And, of course, the famous “Dwight” flasher flyer from “The Office”

The shelf behind me containing random stuff I’ve gotten from eating kid’s meals and ice cream sundaes. And some Yankee Swap gifts. Oh, and I have some windows. That makes my life nicer.

My white board and busted ass bookshelf. And my cool VT light switch cover from Matt, and some random stuff I’ve collected and hung up.

The view of where I sit. I used to use that big ass monitor to do a dual-monitor display, but since I’ve been moving around so much each day, now it’s just there to keep people from having a good look at me.

Finally, the MacBook Pro, my Motorola Q, my 30GB 5G IPod, my noise canceling head phones, and my phone that I don’t ever answer or use. And yeah, that’s Win2k running in Parallels. More on that in a bit.

So that’s where I do my work.

My Mac is setup in a very particular way. The upgrade to Leopard with Spaces has made my life considerably easier. It’s probably easiest to roll through how my Spaces are setup.

Space 1
This is where I use my browser, which is currently Firefox 3 Beta 3, and sometimes Safari 3.

Also running on this space is my “chat” clients. We use Jabber at work, which works nicely with Adium. I’ve also got Twitterrific running on this space.

Space 2
Here’s where my Terminal lives, which is just the default Leopard terminal. Tabbed Terminals make me happy, particularly once I made the default tab switching hot keys to be Command+Left and Command+Right.

It’s all command line and vim and mysql. Good times.

Space 3
Space 3 is where iCal and Mail live. Mail is just downloading my mail from Gmail. iCal is doing some cool stuff. I have most of my life in Google Calendar. iCal subscribes to my calendar feeds from GCal (including my work Outlook calendar–more on that in a second).

With all of my stuff in iCal, I then use the Missing Sync for Windows Mobile to sync my calendars to the previously mentioned Motorola Q (which also connects to my work Exchange server, so it’s almost as a good as a Blackberry).

Space 4
It’s the Windows space! I’ve got Win2k (don’t ask, I had a license lying around) running in Parallels, in Full Screen mode. Parallels runs pretty much just so I can run Outlook (for work email and calendaring) and so I can occasionally test stuff in IE6.

My Outlook runs a plugin called SyncMyCal to sync my Outlook calendar off to Google Calendar (which then gets sync’d down to iCal, as previously described).

Other software that occasionally comes in handy:

• NeoOffice (though it’s slow and bulky and I’d switch if there was a viable alternative)
• iTunes (obvs)
• MarsEdit (for doing this sort of stuff)

That’s how I get my work done. Anything else I should be using?

Oh, and this is all without a cluster. That’s probably the next step. And “Oh: part 2,” myisamchk finished checking 28 million rows. All is good. I’ve copied the data over to the main server and brought it up and everything is happy. Back to MySQL scalability …

First issue: We’ve got too much data

This one was easy to solve. We got rid of it. Sort of. We started archiving off data that we no longer needed chronologically. Old support incidents, logging, anything that had a timestamp that we aren’t looking at gets sliced off into an archive so that the main tables can be as tidy and fast as we can get them. Which for us is like 8GB of data and not fast at all. But it’s better than 12GB of data.

Second Issue: We’ve got too many connections

When you’re small(ish), it makes sense to throw a bunch of dbs on the same server. As you grow, those connections start to swamp MySQL. MySQL starts to get all panicked, and it doesn’t know how to handle all of the people asking for data, so it starts to get sloppy about closing old handles. Then it’s basically like thermal runaway in a transistor. The server can’t close old connections, new ones open up, adding more overhead, and all of a sudden your nice server has 5000 open connections and is hosed. Again, this was a pretty easy one to fix. Bring up a new box, move some databases it to it, and hope that you’ve built your code layer to make that swap pretty easy (ours was). Presto. Now both of your servers are happy.

Third Issue: We lock up the damn tables all the time

We’ve got a lot of customers who are constantly accessing their sites. We’ve got nearly 1000 support agents across the globe who are using our tools to look at customer configuration to make sure there’s no issues. This puts a whole chunk of load and repetitive queries on the database. That’s easily handled.

Except when you add in a bunch of data updates. Agents, customers, new signups adding and editing data in the database. All of a sudden those hundred pending SELECT statements are stuck because one big select locked the data when an UPDATE came in. Now you’ve got a bunch of web users who think your stuff is slow and/or broken. We’ve tried to attack this in a few ways:

1. Fix your queries — We watch our slow queries and try to make them faster. We look at our most often called queries and try to make them faster. Sounds simple, but bad queries are the biggest cause of problems.
2. Add indexes — This goes with “fixing” queries. Add indexes and make sure you’re queries use them.
3. Perform less queries — Can you cache your data? Can you make less queries and do more in your language of choice? Can you make your users smarter (maybe without them knowing) about when they need to request data? Do it. The less queries you have, the more likely you won’t lock things up.
4. Split your reads and your writes — If you can split your reads and writes at the code layer, then you can shuttle reads off to one (or more boxes) and writes off to the primary box, and you should lock up a lot less. We accomplished this by having a couple of boxes replicate the main database, and having one of our smart engineers subclass Perl::DBI to look for SELECT statements and swap the database handle over to the read replica. It helps more than you might thing (but it’s not a silver bullet).

Most of this sounds like common sense. It is. But it still matters. We’re trying to do a lot with a little, and every ounce of performance you can squeeze out matters, when your users are super demanding and will use any slowness as an excuse.

There are some other things we should and probably will try:

• Denormalization to bring data back together and cut down on costly joins
• Sharding to split our data up into smaller chunks and this cut down on long table scans and huge indexes
• A real MySQL cluster to optimize reads and writes and spread traffic out to many nodes

I wish I knew more. I’m still barely up the curve compared to some of the engineers and admins I work with. Thankfully, they’ve been able to keep our many million row tables (and many GB data and index files) humming along with few interruptions.

Advertising on the Web

There’s lots of advertising on the web. The biggest web company in the world (Google, ever heard of them?) generates pretty much all of their revenue from text (and now some image ads) based on the context of your web searches, email, or web page content. Big media sites like ESPN or ABC or NBC generate some revenue and awareness through the old late 90s staple of the banner ad. Blogs, podcasts, and video sites get in on the action with pre/post-roll ads, typical interstitials, and sponsors. Tons and tons of ways for sites to generate some income, but they’re all pretty much based on getting a large number of eyeballs.

With the recent growth of blogs, forums, and just the general smaller sites run by individuals rather than corporations, folks have wanted to cash in on some of that free internet money. But banner ads and AdSense cash really don’t work too well unless you gets loads of traffic. Now, granted, there’s tons of ways to do that (which is why a bunch of the junk that fills up sites like Digg and Reddit these days are obvious linkbait bullshit attempts to generate lots of traffic), but the internet in general–through better algorithms and crowdsourcing and such–has gotten pretty good at weeding those things out. Besides, only a few linkbaiting attempts can work in a given time period, so even this is not a surefire way to get yourself that sack with a dollar sign on it.

Affiliate Marketing: Good

This gave way to a new niche: affiliate marketing. In all honesty, this started a while back with Amazon. And, to this day, Amazon’s method isn’t really bullshit. Amazon’s affiliate system and basically a way for people who review stuff or talk about different things they want to link to those items on Amazon and get a kickback if someone purchases through their link. Everybody wins in this situation; the buyer gets an object they wanted, the site owner gets a few bucks back for setting up Amazon and the buyer, and Amazon gets to sell an object.

Plus Amazon gets some search engine love from having lots of sites link to them.

I’ve used the Amazon affiliate system when doing my not-very-often-update podcast. I’ve never made a dime, but that’s because I don’t do it very often and I’m not exactly linking to highly sought after stuff. But disclosure is important, because it’s the lack of disclosure is a big reason that affiliate marketing is currently the <insert your horrific disease here> of the interweb.

Affiliate Marketing: Bad

Somewhere along the way, the affiliate stuff got bastardized. It was, of course, inevitable. We live in a world of pyramid and get rich quick schemes broadcast in half-hour increments on late night TV. But the internet’s version is far more nefarious. The BS seemed to start in earnest with BzzAgent, a Boston-based marketing company that paid “agents” to go around talking up products–products that the agency had been paid to promote. There was no disclosure; “Hey, I’ve never actually tried this product! I’m getting PAID to tell you that I think it’s awesome!” Of course, BzzAgent took some heat for their arguably deceptive marketing. It was not intentionally deceptive, but they were implicitly offering people incentives to be deceptive. After some bad press and some backlash, BzzAgent claims to be all about the disclosure.

Affiliate Marketing: Worse

Similarly, a company called PayPerPost sprung up. Here’s a group that will pay you to write about a product, service, or other site, right on your own website! You write up a few paragraphs, throw in a few links, and you make some money. The sponsoring company gets some search engine juice and some good word of mouth. PayPerPost gets some money for bringing the two parties together.

Not so different from the Amazon model, right? Sure, except that there was no required disclosure that the post was, basically, just a paid advertisement. Posts from PayPerPost folks weren’t required to be tagged as advertising, the way that those fake magazine articles are. The writer never even needed to try or use the product they were writing about. It was obvious to everyone what was going on: blatant link buying in an attempt to game the search engines.

(For a more complete story on PayPerPost, try TechCrunch.)

So what’s the difference between this and the Amazon model? The end-user. The buyer. They’re getting hosed in that they’re just the commodity being traded in the middle. Taking someone to Amazon, where they’re then exposed to any number of other reviews for the product in question (which, by the way, is almost always a consumer product that’s a tangible good) is incredibly different than linking them to a web hosting company (more on that later) or some other digital good that is not quite as easily identified as something a user does or doesn’t want.

Particularly when all of the reviews for said product are paid for, and thus biased, by the aforementioned affiliate system.

So BzzAgent and PayPerPost started paying people to write about products and services, without a requirement of disclosure, and in many cases, without actually even trying out the product they were promoting. Sure, they were just trading on their online identity–burn people enough and you’re opinions are worthless. That is, of course, unless you can create endless domains and identities. The two companies were rightfully shat upon by the honest folks on the web. Both have started to talk about honest disclosure and transparency in attempt to stay relevant and to ensure their clients don’t run away for fear of being painted with the same dishonest brush.

It hasn’t worked for PayPerPost, whose business was rightfully crippled by Google when Google basically dropped the rank of any site found to be working with PayPerPost. If PayPerPost’s business is as honest as they claim it is, this wouldn’t have mattered. The paying companies would still be lining up to get reviews and links. They’re not. They wanted search juice. And that’s not for sale, well, not through PayPerPost, at least.

Affiliate Marketing: The Drizzling Shits

Which brings me to my biggest pet peeve, and the one that hits closest to home: bullshit web hosting review sites. There’s tons of them. They claim to review web hosts. They don’t. They rank sites based on who pays them the most money per hosting sign up. It’s, quite frankly, a pox on the hosting industry. Each web host offers the affiliate a little more money. In return, the affiliate gives them good links for SEO, some traffic and new sign ups, and a couple of web STDs.

From some of our internal research, somewhere around 20% of all sign ups that come through affiliates are fraudulent. Most of them have a life span significantly shorter than a typical sign up. Many of the sign ups that make it through the front end fraud checks are still BS accounts. They sign up, collect the affiliate fee, and then cancel. With most web hosts, if you did that 10 times a day, you’d make in the neighborhood of $100k a year.

I’m not kidding.

Why is this so bad? Again, it comes down to disclosure. None of these sites reveal they’re doing this for pay. Most of them layer some arbitrary, made up review score on top of their listings, depending on which host is paying the most that month. The affiliate doesn’t care that it’s slimy–they’re getting paid. The web host doesn’t care that it’s slimy–they’re getting new “real” hosting accounts. Who cares? The actual honest person who did hit Google or their search engine of choice to look for a web host to open a blog or a place to host their pictures of their grandkids. They find a review site, sign up with the top rated host (“oh my, this host is rated the top on ten different sites!”), and then find out it’s a completely crappy host. The poor grandma doesn’t realize that ten different review sites were all run by the same person/group/company. She didn’t realize that the top host was paying these affiliates so much because their service is so bad they’re hemorrhaging customers.

I’ll admit, my company pays affiliates. Slimy ones, at that. We’re not hemorrhaging customers. We’ve actually stepped up our game, I think, and have started to deliver a better hosting experience for most of our customers. But growing organically by word of mouth isn’t good enough for us, so we put on the full body web condom and deal with the underbelly of the internet.

It’s disgusting and immoral and we shouldn’t do it. Many of us have made that case. But, unfortunately, the dollars trump us. So we build in workarounds and special rules to pay off certain affiliates to make sure they get the conversions they want so they’ll keep sending us traffic. And keep linking to us.

Affiliate marketing isn’t inherently bad. But, as with anything, when you mix it with the internet, it ends up being more bad than good. It’s the drizzling shits of the internet.

Soon enough, Google will step up and kill this trend. And it’ll be a great day when we can focus on stuff that matters and not spend thousands of man-hours building search algorithms to weed out fake sites, building fraud detection to weed out the fake sign ups, and trying to convince ourselves that just because other folks are doing it, we need to do it to keep up.

Yuck.

Save Us Obi-wan

Dear Matt Cutts,

Can Google please fix the fake review sites? It would be awesome.

Thanks.

Your pal,
The Interweb

Affiliate marketing is everywhere now. Google it. You’ll find hundreds of blogs devoted to how to get a spammy, content-less site ranked high in the search results, get people to click your links to generate conversions, and how to basically make money being dishonest. Granted, all marketing is somewhat dishonest–promote the good stuff, hide the bad stuff. But when it’s a first party doing it, you know to take what they say with a grain of salt (which is why good companies are transparent and talk about their occasional foibles … it makes the marketing spin look less spinny). When a supposed neutral third party is hiding the fact that they’re making money off of their “review,” it’s not easy to discern that. It’s ugly and stupid and dishonest. And it makes people loads of money.

Again, it’s why affiliate marketing is the drizzling shits.

Examples

Just in case you’re wondering, here’s what a bullshit review site looks like. I shouldn’t claim this to be authoritative. I don’t know with 100% certainty that these are fake review sites. But they fit the mold. They cloak their affiliate links, bring you over to the web host with an affiliate cookie, and have surprisingly similar reviews. I won’t link to them, but you can paste them into your address bar.

http://www.best-webhosting2007.com/

http://www.web-hosting-review.toptenreviews.com/

http://www.web-hosting-reviews.org/

http://www.web-hosting-top.com/

http://www.webhostingtoplist.com/

http://www.webhostingfever.com/

http://www.websitehostingreviews.com/

http://www.100best-free-web-space.com/