That Not So Fresh Feeling

I really, really do. I despise it. Over the past few weeks, we’ve been battling in our hosting environment to come up with solutions to a myriad of issues with ColdFusion and how it scales. Now, I don’t think that ColdFusion was really designed to be used in a shared hosting environment, with hundreds of sites and thousands of hits per second throwing traffic at it. It certainly doesn’t scale well enough.

Now, I know there’s a ton of heavily trafficked sites that use ColdFusion. It is possible for it to scale. It’s just not possible for ColdFusion to scale when the code in question isn’t specifically designed to do so. We’ve got thousands of customers spread across a handful of servers, and they’ve all written code that, well, let’s say it’s not professional. It’s all basic “open database, run query, output data” code, 99% of the time against an Access database.

75% of the time, ColdFusion handles its end of the bargain pretty well. We’re running IIS6, Windows Server 2003, all the normal stuff. The problems end up being two fold:

1. ColdFusion still uses ODBC
2. ColdFusion uses Java

Our two basic problem scenarios are cases where we get a lot of concurrent database queries and ColdFusion’s ODBC service simply decides it can’t handle it. Rather than fail gracefully (allowing us to catch the stopped service and restart it accordingly), the service just keeps running, but simply doesn’t do anything. It actively rejects all TCP/IP database connections, and leaves customer sites basically dead in the water.

We’ve worked around the ODBC issue by having a test page that makes a database connection, which we query every X seconds, determine if the ODBC service is still working, and if not, we actually kill the process and restart it. Obviously, we’re adding a little overhead to the box by doing this (not a ton, but it’s overhead) and we’re not really solving the root cause, which is that the ODBC service doesn’t behave the way a normal Windows service should.

This issue is sort of hinted at in this ColdFusion TechNote, but there’s no solution other than to limit connections, which is not easy to do in a shared hosting environment (well, not in any meaningful, cost-effective way). Not to mention the fact that using ODBC and DSNs is pretty antiquated and I’m not sure why ColdFusion hasn’t deprecated it completely in favor of DSN-less connections (which are not only more reliable and have marginally better performance, but are far far far more portable).
The second issue is simply that ColdFusion uses Java. There’s nothing inherently wrong with Java (just as there’s nothing inherently wrong with .NET or any other run-time environment). They all have their places. This just isn’t a place that Java works well. The Java process gets progressively larger because customers don’t write applications that really care about garbage collection or optimization. They’re not homogenized. There’s hundreds or thousands of very different applications, that instantiate different objects, and then the Java garbage collection has to try to figure out exactly how to handle it. Sometimes it works well. Other times it thrashes the CPU usage up to 100% and causes the box to simply hang out while it decides what to do.

We’ve been mucking with ColdFusion and JRE settings to work around this, and we’ve basically mitigated this problem through trial and error, and also some monitoring that will push traffic over to another server if the main server is “hung” and then route traffic back again when things are back to normal.

It’s taken us a few months to get to this point, a few calls to Adobe, and a whole bunch of trial and error. Each time we solve one problem, another pops up that needs investigation and resolution. Having now played with ColdFusion for about a year, I can safely say that I would never pick it over Perl or PHP, but I can see why someone might select it over ASP or .NET, which have slightly more difficult methods of connecting to and handling database info.

Still, I despise ColdFusion. It encourages its users to do stupid things (I cannot even tell you how many customers I’ve had to inform that their code doesn’t work because they’re using a reserved SQL word as a variable or table name — something that happens far less frequently with ASP or PHP users). It relies on a bunch of server-side technology that renders its usage fairly non-portable. It’s got a horrible design for IIS, leading to cases where ColdFusion dying can cause IIS to stop serving HTML pages on the box (thanks wildcard ISAPI filter!). In a nutshell, ColdFusion has pretty much been the bane of my existance for a few months now, but we seem to have it handled at this point. If I ever ran my own hosting company, I’m 99.999% sure I’d never offer ColdFusion, at least not without charging a significant premium.

I despise ColdFusion.

I’m taking a couple days off from work and heading up to the beach for the day with some folks. Should be a nice break from the ColdFusion-rich days I’ve been spending at work. We’re working on a project to take ColdFusion users from being spread out across our Windows servers and move them to their own Windows servers, where poorly written code can’t take down other, non-ColdFusion pages.

I’m not a big fan of ColdFusion. I can understand why people would use it if they’re not particularly skilled developers, but once you know enough to use ColdFusion well, it seems like you’d want to use ASP, Perl, Python, Ruby, PHP … something …. anything else. ColdFusion runs through Java, so it tends to be slow when you’re running it through IIS. Making things worse, we’ve discovered that ColdFusion’s default JDBC-ODBC bridge is pretty much crap. When you get 12 concurrent database queries, the ColdFusion ODBC service dies. But not gracefully, it gets stuck in a state where it can’t be stopped or started. The box has to be rebooted.

Why does this suck royally? First, because ColdFusion users (at least those using our service) tend to write really crappy code without closing their queries and sometimes running queries within queries, and they can hit that 12 concurrent database queries pretty quickly. But the bigger gotcha, the bigger kick in the junk, is that ColdFusion is installed as a wildcard script map in IIS. That is every single page request, ColdFusion or not, goes through ColdFusion for ColdFusion to decide whether or not it wants to handle it. So when ColdFusion dies, NO PAGES GET SERVED FROM THAT BOX AT ALL. It’s really quite annoying.

Yes, there are some things we could do to mitigate it. The logical one would be to remove the wildcard script map, but that actually breaks ColdFusion (some wonderful work you’ve done their, Macromedia/Adobe).

So, we’ve actually decided to segment ColdFusion users and use a native JDBC driver rather than the ColdFusion JDBC-ODBC bridge. I’ve spent the last few weeks of my life on this, moving customer sites, testing, writing Perl code to automate the process. It’s been fun taking existing ColdFusion DSNs and recreating them in the new format. It’s been more fun finding out the various little things that the new JDBC Microsoft Access driver doesn’t support that the traditional ODBC driver does. (Hey, for some ridiculous reason you’ve got an Access replicated table? Fantastic, the JDBC driver won’t read it. Hey, you’re using RND in your query? Fantastic, the JDBC driver doesn’t support it. Hey, you’re using a raw ‘Yes’ in your query to match a checkbox column. Fantastic, change your query to != 0 to pick up the positive values.)

It’s been a long few weeks, but it’s gone pretty smoothly, all things considered. But I hate ColdFusion, and I hate Access.

Thus, I’m off to the beach. Where there will be no ColdFusion. And no Access. Just my iPod, newly loaded up with all of the episodes of the Band in Boston podcast, as pointed out by Bostonist. I’ll be back soon feeling refreshed and ready to deal with more ColdFusion fun.