Back when I worked at On-Site, years ago, I screwed something up in my code. I discovered it when everything stopped working in production.


By “everything” I mean “nearly every action a user could take would just hang.” So that’s not great.

But I learned something important about database migrations. And that’s nearly as good as not crashing production, right?


I’ll tell you what happened with me and crashing our servers, but first let’s talk about database migrations.

My favorite article on salary negotiation of all time talks about “fully-loaded costs” of an employee. The idea is that when figuring what it costs a company to employ an engineer (or whoever) it’s short-sighted to just take their salary and multiply by time. Patrick suggests that "a reasonable guesstimate is between 150% and 200% of their salary" and that the “extra” tends upward as salary does. Of course it depends on benefits and whatnot.

Many people think that’s complete baloney. Specifically, they tend to think that the “extra” is fixed (e.g. $30k extra,) rather than a large and increasing percent of salary.

But when you’re negotiating salary, or otherwise asking, “what does an employee’s time cost a company?” he’s right. Let me explain why.

Breaking down a giant monolithic Rails app (colloquially a “monorail”) is a very hot topic right now. I’ll give you the boring, accepted advice first: extract obvious bits by breaking apart obvious sub-apps, take nontrivial logic and extract it into /lib and/or external gems, take repetitive models/controllers/views and, if possible, extract those into separate Rails-specific gems (“engines”) as well.

There’s compromise in all of that, but it’s all basically solid advice. It only takes you so far, but it’s clearly in the direction of better code.

So let’s talk about some of the newer, less-accepted and dodgier advice, since you’re probably already quite familiar with the previous boring advice :–)

You can break back-end stuff into separate services, then call to them via HTTP from the front end. This is higher-latency but scales better, and lets you build not-specifically Railsy logic outside of Rails, where it belongs. This is an especially good choice if you have components that don’t fit clearly into an HTTP server, such as those that want to run their own threads, processes and/or background jobs. That’s the SOA bit, which is called “microservices” if you’re hip and/or the services are smallish.

Microservices/SOA adds a bunch of interesting issues, though:

1) how do you deploy them? Separately or together? 2) how do you make sure services are running locally on your dev box? 3) the more processes you have, the more likely one died. How do you keep them all running? Same in prod vs dev or different? 4) how do you handle versioning in the API? How do you handle non-backward-compatible upgrades in services? 5) how do the services communicate? HTTP is inefficient, latency-heavy and error-prone, but message-passing is complex and has ugly failure cases.

That’s not all the issues, but it’s a good start. We’re doing the same at my job, so it’s front-of-mind :–)

You’re better off if you look for one piece that’s already pretty separable and make it its own service first. Then slowly extract others after it’s up and working. That lets you ease into a lot of the issues above. It also lets you answer the questions when you need a specific answer, not a general one. One of the big advantages of services is that you can answer these questions in the way that works best for a specific service — which means it’s not always the same answer for each service.

Hope this helps!

I’ve gotten this question several times in several forms, so here’s a typical one, and my current answer…

I’m a mid level Rails engineer with strengths in Ruby, Rails and TDD. I understand OOP and REST, but I am relatively week when it comes to deploying a Rails application. Do you recommend any resources on learning how to deploy a Rails app, grow/maintain its deployed environment, and optimize for your application’s performance and scaling abilities?

In general, not really. This turns out to be a gaping hole in the landscape.

Part of the problem is that there’s not one standard way to do this, and the methods change constantly. You can find tutorials and (sometimes) books on specific, individual tools like Chef, Ansible, Capistrano, Docker and so on, but they’re terrible about providing an overview. You have to already have a good idea of what the top level looks like, which is difficult given that the tools make different assumptions and are for different scenarios, but don’t spell that out.

You can see an example of one way to set it up in my Ruby Mad Science open-source software, but please don’t purchase the associated class. I’m in the process of winding it down.

For pretty much the same reason, it turns out. Deployment is something everybody currently does custom. It’s possible that a Rails-style entrant (and/or Rails itself) will eventually standardize this enough to allow one flavor of deployment instead of thousands or millions of flavors, but we’re absolutely not there yet. Heroku is the current closest.

Certainly look at Heroku if you haven’t already. It’s the only example of a simple standard method of deployment, and it’s the gold standard for “it just works.” It’s also expensive and not very configurable, but it’s still worth looking at.

Right now, people go out and learn by doing, with highly variable results :–(

Occasionally, you’ll see people put up a “why you should hire me” page, such as the one that got Jason Zimdars a gig at 37signals. These pages are effectively sales pages for people, though they read a little differently from a sales page for a book.

(If you don’t believe that a “hire me” page is a sales page, I recommend reading a bit more Patrick McKenzie.)

Most frequently you see pages for less-established entry-level people — people who try using a sales page because they have to, not because they want to.

And I thought, “hey, what if I tried that?” As a great fictitious man once said, "we try things. Occasionally they even work."

So when my last company went out of business, I put a “hire me” page together. I had a little breathing room before missing a paycheck and reasonable savings. Why not try it?

Yeah, Why Not Try It?

When people talk about Google ruling the roost, it’s common to compare them to Microsoft. I’m an old guy, and I remember Microsoft as our overlord. So I find that comparison pretty darn funny.

But if you haven’t been doing this since, oh, call it 2005… That doesn’t necessarily mean much to you. Microsoft? They’re not anybody special any more. Certainly they’re not unusually evil. They’re not especially powerful, even. Yahoo is probably as strong as current Microsoft, and Yahoo doesn’t intimidate people into anything.

What was Old Microsoft like that was so scary?

Old Boys

Microsoft specialized in a few specific things:

  • Packaging and contract-writing rivals to their core businesses (OS, Office) out of existence, often through straight-up illegal pricing.
  • Leveraging their OS and Office monopoly to bundle other products, killing rivals with ‘free’ included versions (e.g. Internet Explorer, leveraged to support their web servers, back when those cost money.)
  • Suing rivals out of existence, often just because it was more convenient than competing
  • Cozying up to small companies, hiring their primary engineers, cloning their product and folding it into Windows — even if the product was stupid in the first place. Amusingly, that’s how we got Clippy.
  • Out-marketing technically superior companies, killing them because they’d been outbid in the marketing channels.
  • Putting in MS-only optimizations into the OS and/or specifically putting in “screw this one other company” sabotage. Remember, everybody had to use Windows back then.

Google is a “Good Old Boys” style company like GM — “what’s good for General Motors is good for America.” In fact, you’ll literally hear Google folks say things like, “what’s good for the Internet is good for Google.”

Now, Google is an ad company and they act like it. But they’re in a wide-open space, they can literally make lots more money by getting more people using the Internet, and nothing they care about is particularly competitive. Blue ocean all the way.

This is much, much better than “carpet bomb potential competitors, eat young companies that might do well, sue and market everybody into oblivion” old Microsoft. Microsoft had a weird persecuted streak that translated into them being the big dog but believing mauling any little dog that they thought might get big.

Which meant if you were a little dog back in the days of packaged software, you had to get past a paranoid mortal rival (Microsoft) just to sell your software — everybody used Windows, so you had to deploy to a Microsoft platform. And I promise you, they played very rough sabotaging some of their competitors with the OS. There was no digital distribution platform. There wasn’t a browser you’d want to deploy to. Just the desktop.

Microsoft didn’t come down on everybody, but when they came down, they came down hard. You were always praying the they didn’t see your market as too profitable or too strategic. Only the biggest companies could fight them — and even the biggest companies couldn’t usually win, just fight longer before they sank.

Paul Graham (figuratively) sang a lovely little ode to the end of that era when he noticed it was over.