Red Leopard

Things don’t change. You change your way of looking, that’s all.
— Carlos Castañeda

Early this spring, I made some big architectural changes in the company’s website. The two most far-reaching changes involved Amazon’s Web Services and Adobe’s Flex product. Sometimes you regret big changes. I only regret having not made the changes earlier.

I admit that I wasn’t always a flex fan. Indeed, I dismissed the flex out of hand in the early days mainly as a response to Macromedia’s steep pricing model. Ouch. Since then, Adobe had bought Macromedia and the pricing model changed several times. I never noticed. Such is the cost of writing something off.

Then I saw Peter Armstrong’s presentation at the Third Annual Silicon Valley Ruby Conference (April 18-19, 2008). Turns out, flex is now free. (uhhhmmm, ok. you will want to buy the developer ide but it’s possible to use the free command line compiler.)

I bought Armstrong’s book and started working through the tutorials. By the time I finished Chapter 6, I knew that I was going to change our webapp from a javascript-driven front-end to an flex-based, actionscript-driven front-end. We deal with a lot of tabulated data. Over a million sets and growing. Some of the datasets are small and some are quite large. Furthermore, each row had numerous event listeners attached. We had plans to add more, add mashups, add slicky bells and wistles. The larger tables were dozens of megabytes large. Which isn’t a problem in and of itself except that…

Javascript sucks.

Well, not really. It’s the browser’s that suck. All of them. We had to back off functionality just to get IE to render. Firefox’s javascript+dom engine (including ff3) would render the tables (if it didn’t crash) but not all the event listeners worked. Safari (web-kit) would render and all events worked but then Apple released the 5-second script warning. We schemed endlessly on how to break the problem up. Mostly we hoped one of the miracle javascript libraries would solve these problems. We knew it was unlikely but we hoped.

I made some side-by-side comparisons on my MacBookPro and actionscript beat javascript hands down. Orders of magnitude faster to load and render. Less code. Consistency between browsers. And finally, actionscript/flex worked. And worked well.

What didn’t work well was our MySQL database. Our million datasets occupied hundreds of millions of database rows in scads of tables and hundreds of gigabytes of disk space. A 3ware RAID10 disk subsystem with master/slave configuration was in complete IO overload all the time. I had Matt from mysql come in to help us unravel our mess. This guy was good. He identified, explained, documented and proposed a prioritized list of solutions.

But I started thinking, why the hell do we even store the datasets in MySQL in the first place? The only time we ever change a dataset is when we replace it. The relational database solution had always been a problem and was getting worse by the day. Thinking like a mathematician, I decided to solve a different problem. Why not just save the datasets in flatfiles and overwrite them when they change? Just store them in flatfiles. Like pictures.

Of course, every solution brings its own problems. And storing a million flatfiles has its problem: you cannot store a million files in a single directory (not on Centos+ext3 you cannot). I recalled reading about mogileFS some time back. It presented what looked like a flat directory in which you could store millions of files in a single directory, distributed the save over multiple machines with redundancy, and so on. Looking at that solution lead me to smugFS. And to Don MacAskill’s presentation on Scalability. Crap! I said. I don’t need to host the flatfile’s at our colocation center. I’ll put them on Amazon S3. And While I’m at it, I’ll build a RESTful API on Amazon EC2 to import the datasets.

And my team has done it. We have completed the move to EC2/S3 and Flex. I couldn’t be happier. Armstrong’s book, Flexible Rails, lit the spark for our migration to Flexible Web Services.

I will write about some of the hurdles we’ve overcome in future posts. Some of the things we’ve solved are (i) building an EC2 image from scratch, (ii) compressing flat files in a format which actionscripts’ ByteArray can uncompress, (iii) using the struts2 REST plugin, (iv) XML parsing using StAX.