As I look into database migration dangers for a book I’m working on, I find a lot of great existing stuff that I had no idea existed.

Thanks, guys!

Andrew Kane of Instacart has a bunch of wonderful guides, including one on "strong migrations" — migrations that run without downtime on MySQL and Postgres. It sums up some wonderful previous articles about downtime-free migrations on Postgres with high data volume, how to create indices without downtime on Postgres, and no-downtime migrations in general.

Some of you may remember Rails Deploy In An Hour, a Ruby deployment class I started selling last year. More of you probably remember Ruby Mad Science, the open-source software that went with it.

I’m not selling the class any more, though I haven’t taken down Ruby Mad Science, and I don’t intend to. But it won’t be getting the maintenance it would want… now that I understand just how much that is.

I wondered, “why aren’t there more good Ruby deployment products out there?” And “why don’t people know more about the books there already are?” And as I learned more I started to wonder, “why aren’t there more Heroku competitors?” And “why did Ninefold stop supporting Rails?”

These are interesting questions. I know the answers better now, and I’m happy to share them. Perhaps they’ll help the next person fix the problem better than I did.

What People Want

It turns out that programmers hate doing deployment. The programmers I’m targeting (hobbyists, early developers, small startup guys) extra-specially hate doing deployment. They just want it done and working with minimal work.

Back when I worked at On-Site, years ago, I screwed something up in my code. I discovered it when everything stopped working in production.


By “everything” I mean “nearly every action a user could take would just hang.” So that’s not great.

But I learned something important about database migrations. And that’s nearly as good as not crashing production, right?


I’ll tell you what happened with me and crashing our servers, but first let’s talk about database migrations.

My favorite article on salary negotiation of all time talks about “fully-loaded costs” of an employee. The idea is that when figuring what it costs a company to employ an engineer (or whoever) it’s short-sighted to just take their salary and multiply by time. Patrick suggests that "a reasonable guesstimate is between 150% and 200% of their salary" and that the “extra” tends upward as salary does. Of course it depends on benefits and whatnot.

Many people think that’s complete baloney. Specifically, they tend to think that the “extra” is fixed (e.g. $30k extra,) rather than a large and increasing percent of salary.

But when you’re negotiating salary, or otherwise asking, “what does an employee’s time cost a company?” he’s right. Let me explain why.

Breaking down a giant monolithic Rails app (colloquially a “monorail”) is a very hot topic right now. I’ll give you the boring, accepted advice first: extract obvious bits by breaking apart obvious sub-apps, take nontrivial logic and extract it into /lib and/or external gems, take repetitive models/controllers/views and, if possible, extract those into separate Rails-specific gems (“engines”) as well.

There’s compromise in all of that, but it’s all basically solid advice. It only takes you so far, but it’s clearly in the direction of better code.

So let’s talk about some of the newer, less-accepted and dodgier advice, since you’re probably already quite familiar with the previous boring advice :–)

You can break back-end stuff into separate services, then call to them via HTTP from the front end. This is higher-latency but scales better, and lets you build not-specifically Railsy logic outside of Rails, where it belongs. This is an especially good choice if you have components that don’t fit clearly into an HTTP server, such as those that want to run their own threads, processes and/or background jobs. That’s the SOA bit, which is called “microservices” if you’re hip and/or the services are smallish.

Microservices/SOA adds a bunch of interesting issues, though:

1) how do you deploy them? Separately or together? 2) how do you make sure services are running locally on your dev box? 3) the more processes you have, the more likely one died. How do you keep them all running? Same in prod vs dev or different? 4) how do you handle versioning in the API? How do you handle non-backward-compatible upgrades in services? 5) how do the services communicate? HTTP is inefficient, latency-heavy and error-prone, but message-passing is complex and has ugly failure cases.

That’s not all the issues, but it’s a good start. We’re doing the same at my job, so it’s front-of-mind :–)

You’re better off if you look for one piece that’s already pretty separable and make it its own service first. Then slowly extract others after it’s up and working. That lets you ease into a lot of the issues above. It also lets you answer the questions when you need a specific answer, not a general one. One of the big advantages of services is that you can answer these questions in the way that works best for a specific service — which means it’s not always the same answer for each service.

Hope this helps!

I’ve gotten this question several times in several forms, so here’s a typical one, and my current answer…

I’m a mid level Rails engineer with strengths in Ruby, Rails and TDD. I understand OOP and REST, but I am relatively week when it comes to deploying a Rails application. Do you recommend any resources on learning how to deploy a Rails app, grow/maintain its deployed environment, and optimize for your application’s performance and scaling abilities?

In general, not really. This turns out to be a gaping hole in the landscape.

Part of the problem is that there’s not one standard way to do this, and the methods change constantly. You can find tutorials and (sometimes) books on specific, individual tools like Chef, Ansible, Capistrano, Docker and so on, but they’re terrible about providing an overview. You have to already have a good idea of what the top level looks like, which is difficult given that the tools make different assumptions and are for different scenarios, but don’t spell that out.

You can see an example of one way to set it up in my Ruby Mad Science open-source software, but please don’t purchase the associated class. I’m in the process of winding it down.

For pretty much the same reason, it turns out. Deployment is something everybody currently does custom. It’s possible that a Rails-style entrant (and/or Rails itself) will eventually standardize this enough to allow one flavor of deployment instead of thousands or millions of flavors, but we’re absolutely not there yet. Heroku is the current closest.

Certainly look at Heroku if you haven’t already. It’s the only example of a simple standard method of deployment, and it’s the gold standard for “it just works.” It’s also expensive and not very configurable, but it’s still worth looking at.

Right now, people go out and learn by doing, with highly variable results :–(