"As God as my witness, I thought turkeys could fly"

The other day at trivia (where we came in 4th out of 25 teams), there was a question about WKRP in Cincinnati, which I remembered, but no one else did. I mentioned to my teammates what may be the funniest sequence from any episode I can remember. Then I went on hulu.com, and lo! there it was.

I figured this was a good chance to try out the Hulu embedded clip thing. So, here's what I think is a damn funny 3 minutes from WKRP in Cincinnati.

(If this doesn't work, I'll fix it. Just trying to do the Hulu embed through MarsEdit.)

Going Hollywood Isn't All That Exciting

They're filming a movie near my office. It's affectionately referred to by IMDB as "Untitled Kevin James Comedy." I'm betting on it being a huge hit for a couple of reasons:

• It is written by *and* stars Kevin James
• It is also written by Salem the Cat
• It also stars Jayma Mays
• It is being filmed in Burlington, MA. At the Mall.

I smell a hit.

Anyway, they've apparently set up a production office right behind our building, and as I was walking over to talk to someone on the other side of the office, I saw some guy go zipping down a zip wire. I tried to get my camera in time, but missed any other awesome festivities. I did, however, take some awesome pictures of the amazing movie production set.

A ramp and a crane. No idea what they're for, but they're obviously super cool.

Some dudes and some metal stuff that looks like they're about to have a steel cage match.

What movie would be complete without a Segway scooter-thingy.

Sadly, I missed the casting call to be an extra, I think. But that's ok, I'm sure they'll grab me from the Apple Store and decide to make me a leading character, ideally with magic powers and a talking cat (who once tried to take over the world).

Pimp Me A Mac App

I wanna buy a new Mac app.

Things I'm considering:

• Coda -- The all-in-one web dev app is pretty cool, good looking, and I've used the demo to build a couple of pages. It's nice. It's also expensive. It'd be a no brainer at sub $50. I'm close to buying it when it's discounted at $79. It's hard to pull the trigger at $99.
• Twitterrific -- Almost a no brainer at $15, but all it does is remove ads and that just seems kinda wasteful to me.
• Warcraft 3 -- I just kind of want to play an old school RTS like Warcraft or Starcraft.

Sell me on an app or a game. Maybe I'll buy it.

Word To The Wise

Always, Always, Always start your MySQL update query with the limit and the where clause, even if you've done the query a thousand times before.

The one time you screw it up, it will be on a Friday evening, when you're just trying to do one last thing for the day before you head home, and when most of the office has gone home.

Oh, and it'll be a database of 30 million DNS records and you'll manage to wipe them all out.

Yep, that's why I did last Friday. I was stupid and working too fast, and managed to hit the ' and Enter in quick succession, as they sit next to eat other on the keyboard. Caught my mistake when it wiped out "only" 13k records on the master database, but the query had already replicated out and managed to take out the slaves.

My saving grace is that we've got a couple of great folks in our sysadmin group, and they helped me get things back from a backup pretty quickly.

But yeah, that was scary. And not fun. Now I've been on both sides. I've helped people recover from a disaster ... and I've caused a huge disaster. Go me.

Those Who Can't, Teach

Late on Wednesday, I was asked in passing if I would mind basically being the guinea pig for a new training technique we're trying at work. We're growing pretty rapidly, so we're constantly bringing in new folks, folks who don't necessarily come from a web/hosting background (or sometimes even from an internet background). Putting aside for the moment whether or not that's a good thing, the goal is to do some formal "classes" where anyone can come attend an entry level class learning about DNS or mail or FTP or the web. These will be recorded and can be reviewed by anyone later on. We'll do some online tests and those who pass can then take the more advanced classes.

My topics were the aforementioned DNS and mail.

It's an interesting idea, if a bit idealistic. The idea has its own set of problems, but I was a) a bit excited that I got asked to do it, and b) optimistic that it would be at least a little more effective at getting folks up to speed than our current training, which sometimes assumes a base level of understanding and vocabulary.

So, I decided I would try to do it as well as I could, and not wing it like I usually do. I train a good bit as part of my job. Project Managers at my company end up training the rest of the company on how our apps work when we're rolling them out. Candidly, that should be the Product Managers role (and in some cases, they do fill it), but it's just never really been the prerogative of the company to have Product Managers tasked with that requirement.

Regardless, I train a good bit, and I just wing it, which often leads to moments of "well, when we fix this bug, this will work this way ... for now, let's just *pretend* it worked." I'm not terribly fond of my public speaking skills, which include a lot of "umm" and "ahh" and "well", but I'm pretty proud of my ability to react on the fly and adjust when things aren't going well. I also introduce a lot of humor (well, attempts at humor) in an attempt to distract from my sometimes below par speaking skills and to help at least keep people engaged in what can be dry material.

(Note: by humor, I should mean referencing 80s TV shows, talking about how swirling your mouse makes things run faster, making pubs. I don't mean that I'm actually funny. I'm not.)

This time needed to be different. It needed to be a legitimately strong presentation, with structure and flow. Of course, I didn't have a whole lot of time to prep (given that I still needed to do my real job). I spent Wednesday night on the treadmill at the gym going through the basic premise of what I would do. I spent Friday morning writing some bulleted notes which I turned into a NeoOffice presentation.

The training started at 2pm and, other than talking a bit too fast, I think I pretty much nailed them. Since I had notes and a basic flow, I didn't spend much time wondering where I was going. I tied the mail presentation into DNS. I dropped out of the presentation to take people through a Terminal-based SMTP and POP session. I even threw in the good old AOL "You've Got Mail" sound. 2 hours of training, minimal screw ups, questions answered on the fly. I think for the first time in a very long time, I was actually not embarrassed by my speaking abilities or my attempts of humor to cover them up.

I'm kinda lucky to have such a varied job (programming, managing, training, documenting, and more).

Wow, this was a complete ego-stroke post here. I'll write about something more interesting next.

How I Get Stuff Done

I've been meaning for a while to sort of document how I get stuff done at work. It was just over a year ago that I bought my MacBook Pro. Within a week or so, I started using it at work. Probably within the first month, I'd completely moved to my MacBook as my sole work machine. After a year, and particularly since the upgrade to Leopard, I've kind of worked out how I get stuff done.

Let's start with my environment.

The front wall of my office with pictures taken by a former co-worker. And, of course, the famous "Dwight" flasher flyer from "The Office"

The shelf behind me containing random stuff I've gotten from eating kid's meals and ice cream sundaes. And some Yankee Swap gifts. Oh, and I have some windows. That makes my life nicer.

My white board and busted ass bookshelf. And my cool VT light switch cover from Matt, and some random stuff I've collected and hung up.

The view of where I sit. I used to use that big ass monitor to do a dual-monitor display, but since I've been moving around so much each day, now it's just there to keep people from having a good look at me.

Finally, the MacBook Pro, my Motorola Q, my 30GB 5G IPod, my noise canceling head phones, and my phone that I don't ever answer or use. And yeah, that's Win2k running in Parallels. More on that in a bit.

So that's where I do my work.

My Mac is setup in a very particular way. The upgrade to Leopard with Spaces has made my life considerably easier. It's probably easiest to roll through how my Spaces are setup.

Space 1
This is where I use my browser, which is currently Firefox 3 Beta 3, and sometimes Safari 3.

Also running on this space is my "chat" clients. We use Jabber at work, which works nicely with Adium. I've also got Twitterrific running on this space.

Space 2
Here's where my Terminal lives, which is just the default Leopard terminal. Tabbed Terminals make me happy, particularly once I made the default tab switching hot keys to be Command+Left and Command+Right.

It's all command line and vim and mysql. Good times.

Space 3
Space 3 is where iCal and Mail live. Mail is just downloading my mail from Gmail. iCal is doing some cool stuff. I have most of my life in Google Calendar. iCal subscribes to my calendar feeds from GCal (including my work Outlook calendar--more on that in a second).

With all of my stuff in iCal, I then use the Missing Sync for Windows Mobile to sync my calendars to the previously mentioned Motorola Q (which also connects to my work Exchange server, so it's almost as a good as a Blackberry).

Space 4
It's the Windows space! I've got Win2k (don't ask, I had a license lying around) running in Parallels, in Full Screen mode. Parallels runs pretty much just so I can run Outlook (for work email and calendaring) and so I can occasionally test stuff in IE6.

My Outlook runs a plugin called SyncMyCal to sync my Outlook calendar off to Google Calendar (which then gets sync'd down to iCal, as previously described).

Other software that occasionally comes in handy:

• NeoOffice (though it's slow and bulky and I'd switch if there was a viable alternative)
• iTunes (obvs)
• MarsEdit (for doing this sort of stuff)

That's how I get my work done. Anything else I should be using?

Bad Week. Really, Just a Bad Wednesday

So, last week:

• I worked pretty late every night on stuff that, for the most part, has nothing to do with my job
• Worked late enough on Wednesday, that I had to drive all the way to the Garden to not miss the entire Celtics game
• Rather than parking at the T and paying $3, I drove around and finally parked at the Government Center garage for $20
• I missed the first half of the Celtics game
• I then spent 40 minutes in park waiting to get out of the Government Center garage
• On the drive home, because I parked at Government Center, and not Sullivan Square (like I normally do), I took 93N home
• 93N was some of the worst road conditions I've ever driven in, leading me to skid towards parked traffic in front of me
• To avoid said parked traffic (already slowed by an accident), I ended up cutting my wheel as hard as possible, resulting in my car doing a 540 degree spin on 93N, catching the rear left bumper against the concrete median and sitting facing into oncoming traffic who had as difficult a time stopping as I did

So not the best week. Topping it off, my poor girlfriend had all of her wisdom teeth out and has been in pain for the better part of 4 days.

Here's to Monday bringing on something better.

(For what it's worth, it isn't all bad. I went to a fun party, got to see the second half of the Celtics win over the Cavs, had a moral victory over a douchebag trying to cut me off in the garage, and got to spend a lot of time with the recovering girlfriend--including watching Blade Runner for the first time. And my car is in pretty good shape, all things being considered.)

Pop-under Ads from Digg?

Am I the only person getting pop-under ads from Digg? I don't think it's malware (it's on my Mac). I don't think it's another site (I've only got Gmail open).

I'm not getting it on every page load, but it's definitely happening. I'm guessing that it's being caused by a rogue ad that's getting brought it by the many ad services Digg uses.

I've only been getting it when I leave Digg up for a while. Lately, Digg pages never seem to complete loading--Firefox just spins and spins trying to load the last couple of elements of the page. I wonder if the two are correlated. Of course, after getting two ads in about 15 minutes, I haven't gotten another one since.

Thoughts on MySQL Scalability From a Certified MySQL Moron

While I sit here and wait for myisamchk to finish and tell me that the table that various folks have spent the better part of the day trying to restore is either healthy or once again dead (how's that for a run-on sentence), I wanted to dump out some of the things we've done to try to make our MySQL backend scale. It's not been pretty, but given that it's strung together with some Perl, some MySQL, and a bunch of paper clips, I think the folks around me have proven themselves brilliant (I just sit around and pretend to know what's going on).

Oh, and this is all without a cluster. That's probably the next step. And "Oh: part 2," myisamchk finished checking 28 million rows. All is good. I've copied the data over to the main server and brought it up and everything is happy. Back to MySQL scalability ...

First issue: We've got too much data

This one was easy to solve. We got rid of it. Sort of. We started archiving off data that we no longer needed chronologically. Old support incidents, logging, anything that had a timestamp that we aren't looking at gets sliced off into an archive so that the main tables can be as tidy and fast as we can get them. Which for us is like 8GB of data and not fast at all. But it's better than 12GB of data.

Second Issue: We've got too many connections

When you're small(ish), it makes sense to throw a bunch of dbs on the same server. As you grow, those connections start to swamp MySQL. MySQL starts to get all panicked, and it doesn't know how to handle all of the people asking for data, so it starts to get sloppy about closing old handles. Then it's basically like thermal runaway in a transistor. The server can't close old connections, new ones open up, adding more overhead, and all of a sudden your nice server has 5000 open connections and is hosed. Again, this was a pretty easy one to fix. Bring up a new box, move some databases it to it, and hope that you've built your code layer to make that swap pretty easy (ours was). Presto. Now both of your servers are happy.

Third Issue: We lock up the damn tables all the time

We've got a lot of customers who are constantly accessing their sites. We've got nearly 1000 support agents across the globe who are using our tools to look at customer configuration to make sure there's no issues. This puts a whole chunk of load and repetitive queries on the database. That's easily handled.

Except when you add in a bunch of data updates. Agents, customers, new signups adding and editing data in the database. All of a sudden those hundred pending SELECT statements are stuck because one big select locked the data when an UPDATE came in. Now you've got a bunch of web users who think your stuff is slow and/or broken. We've tried to attack this in a few ways:

1. Fix your queries -- We watch our slow queries and try to make them faster. We look at our most often called queries and try to make them faster. Sounds simple, but bad queries are the biggest cause of problems.
2. Add indexes -- This goes with "fixing" queries. Add indexes and make sure you're queries use them.
3. Perform less queries -- Can you cache your data? Can you make less queries and do more in your language of choice? Can you make your users smarter (maybe without them knowing) about when they need to request data? Do it. The less queries you have, the more likely you won't lock things up.
4. Split your reads and your writes -- If you can split your reads and writes at the code layer, then you can shuttle reads off to one (or more boxes) and writes off to the primary box, and you should lock up a lot less. We accomplished this by having a couple of boxes replicate the main database, and having one of our smart engineers subclass Perl::DBI to look for SELECT statements and swap the database handle over to the read replica. It helps more than you might thing (but it's not a silver bullet).

Most of this sounds like common sense. It is. But it still matters. We're trying to do a lot with a little, and every ounce of performance you can squeeze out matters, when your users are super demanding and will use any slowness as an excuse.

There are some other things we should and probably will try:

• Denormalization to bring data back together and cut down on costly joins
• Sharding to split our data up into smaller chunks and this cut down on long table scans and huge indexes
• A real MySQL cluster to optimize reads and writes and spread traffic out to many nodes

I wish I knew more. I'm still barely up the curve compared to some of the engineers and admins I work with. Thankfully, they've been able to keep our many million row tables (and many GB data and index files) humming along with few interruptions.

Awakened By Awaken

So, after yesterday's post about the move to shareware of Aurora (which, again, I don't begrudge the dev that--it just doesn't fit into my plans), I got an email from Jerry of Embraceware, the creators of another Mac alarm clock app called Awaken.

Jerry had a good deal for me: a free copy of Awaken.

That's pretty amazing. I'm a small little blog, written by a web dork. My readership is pretty small. But Jerry is the smartest of marketers. What better way to get the word out about your product than to get it into someone's hands who'll try it, write about it, and probably evangelize for it (just based off the fact that it was such a great out of the blue offer).

So I installed Awaken last night, and I woke up to this morning to Bloc Party. It works as advertised, has a pretty nifty interface, and has one feature that Aurora doesn't: it doesn't require you to turn off your screen saver password or turn on auto-login. Both of those lower the security of your Mac (particularly if you're using a laptop, like me), and it always bugged me about Aurora.

Now, with Awaken, even though my machine was locked, the alarm clock turned on and work me up, allowed me to snooze it or turn off the alarm, and then I could go about my daily business.

I like that. A lot.

It's a pretty app too (especially in full screen mode when you wake up). Here's a taste (from Embraceware's flickr account):

Awaken rocks a good bit. I'm probably going to move to it as my alarm clock--it'll be a whole lot less abrasive than my blaring alarm clock from 1988 (that my girlfriend despises). And then I'll probably start using it as my sleep timer too, since it'd be pretty cool to toss a movie into iTunes and have it shut down my Mac when it's done.

Jerry's done pretty darn good marketing here. I would probably have not even considered Awaken before, but now I'm evangelizing it. I love the internet.

You can check out Jerry's blog for more info on Awaken.

← Older Newer →