In this series I’ve built out a Rails app from scratch, deployed it and I’m this close to getting it actually useful.

Will today be that final bit to hit “useful?” I don’t know as I write this, but it’s getting closer…

Last week we built out subscription buttons, profiles for managing subscriptions and a lot of other stuff that doesn’t actually send any email. Let’s fix that.

But first let’s talk a bit about email.

Transactional and Recurring Email

With email, a lot of your worries as a developer are about legality. How do I make sure I’m not annoying anybody or, worse, breaking laws in how I annoy them?

One answer is to use a service like MailChimp that makes it easy and permanent for people to unsubscribe, and that tries hard to require you to meet all your legal obligations. Those services are designed for marketing and promotional emails, which usually have the strongest legal restrictions. MailChimp is who I use for my email list you see below and they’re great, especially with double opt-in. With work you can even get them to stop tracking opens, as I have for my regular newsletters but not (yet) my automations.

However, that’s not what I’ll be using for RubyMadScience email reminders.

The problem with an awesomely-responsible service like MailChimp is, what do you do when the laws on marketing emails are in conflict with site usability? If somebody hits “unsubscribe” at the bottom of one of your emails, they don’t really expect that password reset will never work again. But they did just tell you to never email them again. So which do they want? Legally speaking, MailChimp says “they told you never, and that means we will never let you email them again.”

That’s very responsible. It’s also rough in terms of password resets or (for a commercial site) receipts, invoices, confirmations… Those latter things are called Transactional Email, which basically means “email in response to a direct action the user just took.” For instance, a request for a password reset or a receipt for a purchase or a confirmation of an action.

Transactional emails are in a different legal category than marketing emails. But they’re hard to tell apart. Luckily, I don’t think RubyMadScience is going to be emailing anything that looks much like marketing or promotion.

In fact, “hey, I asked you to email me weekly about this topic until it’s finished” isn’t really transactional or promotional. It’s in a weird grey area. But I don’t think it wants the restrictions that a newsletter tool like MailChimp would entail.

By picking the all-transactional route, I’m also losing out in terms of delivery. Spam filters love the MailChimp approach where they know who’s on your list, carefully get opt-in and repeatedly deal with the exact same recipients. If I thought RubyMadScience was going to be a money-making product I would look carefully at a newsletter-type service and, even if it was painful, use it for the email reminders. I would get better deliverability, better speed

Logistics: Adding Email with Sendgrid

I expect RubyMadScience to start with few users. Daily/weekly/etc email reminders don’t need to be delivered quickly. Even things like password resets can have a nice long time limit.

That means I can use very cheap email delivery via SendGrid. I may not even exceed its free monthly tier. Free is a good price for a service I don’t earn money from. If I’m ever looking at more than 12,000 emails per month, I’ll find a way to make money off it. Cheap email is still good (and cheap.)

Getting SendGrid to actually send took a bit of doing, though. The configuration they give for Ruby on Rails is basically right, but:

• You should turn on exceptions on email send to make sure you see problems when they happen
• You need to tell SendGrid it’s okay to send from certain addresses.

These things are individually quite sensible. Put together, they can easily result in all your email being silently ignored because you didn’t realise what needed to happen, or that you haven’t done it.

I’m going to need to figure out how to keep email exceptions from being swallowed, because that’s not okay. But just turning on exceptions will make transient email timeouts give 500s from my site. Also not okay. This is going to be one of those things I don’t fix perfectly on the first try — I can already tell.

Also, email is one of those things it’s possible to screw up really badly in a way that makes people angry. So, y'know, there’s that.

Frankly, I’m glad there aren’t more of you reading this. My own obscurity is probably my best protection from over-engineering right now. I want to build a huge gated outgoing email-trap to let me do full review before anything gets sent. But that’s only a danger if anybody ever uses the service, so onwards I go.

The courage of a software developer is in building features thinking, “somebody will test this before it goes out.” The courage of a QA engineer is in thinking, “somebody else wrote this, I just have to record any problems I find,” and the courage of an Ops engineer is in thinking, “I’m just babysitting the awful thing somebody else wrote.” Between them they can often be entirely fearless through a whole feature-release cycle, just by everybody getting to blame somebody else.

One problem here is that nobody else could possibly be to blame for any feature I write, test, release and maintain for myself.

Turning Subscriptions Into Reminders

Excellent. I can write a simple Sidekiq job to make rows in this table from people’s reminder settings. I’d like to make sure that daily and monthly jobs arrive in the same email and that the behaviour is generally sane… and now that I’m working out what all that means, let’s do the simple thing first. Here’s the simple thing:

# No reason that the User email reminder logic can't live
# a simple, separate PORO.

# For now, this assumes all sends happen at roughly the same time of
# day, so we can mostly count in whole days. Convenient!
#
# Can that condition fail? Absolutely. Bounced mail and server downtime
# are just the first two reasons that come to mind.
module ReminderCalculator
    # Pass in a hash 'topics' of the form:
    #    "topic_name" => {
    #        frequency: "weekly",      # Or "daily", "monthly" or "none"
    #        last_reminder: Time.now,  # Whenever last reminder was for this topic
    #    }
    def topics_to_remind(topics_hash, send_time)
        topics = topics_hash.dup
        topics.delete_if { |k, v| v[:frequency] == "none" }

        topics_to_remind = topics.keys.select do |topic_name|
            topic = topics[topic_name]
            approx_next_reminder = if topic[:frequency] == "daily"
                topic[:last_reminder].advance(days: 1, hours: -2)
            elsif topic[:frequency] == "weekly"
                topic[:last_reminder].advance(weeks: 1, hours: -2)
            elsif topic[:frequency] == "monthly"
                topic[:last_reminder].advance(months: 1, hours: -2)
            else
                raise "Unknown frequency #{topic[:frequency].inspect} for topic #{topic_name.inspect}!"
            end

            # Return true for this topic name if it's time to send again
            approx_next_reminder <= send_time
        end
    end

    # This takes a list of topic_ids and returns a mapping
    # of those which, after step_completions, still have at
    # least one unfinished step.

    # This method looks up Topics by ID, which I feel a
    # little odd about. The rationale is that it avoids
    # database objects, but Topic is loaded from a file.
    # That's true, and test-relevant, but it also mixes
    # levels of abstraction. Separating this logic from
    # its models has been messy and could likely be done
    # better.

    # Step_completions is a list of items that look like
    # UserStepItems - they have accessors for topic_id and
    # step_id, and if passed in are assumed to correspond
    # to "yes, this step is complete (or skipped.)"
    def next_step_by_topic_id(topic_ids, step_completions)
        topic_steps = {}
        topic_ids.each do |topic_id|
            topic = Topic.find(topic_id)
            topic_steps[topic_id] = {}
            topic.steps.each do |step|
                topic_steps[topic_id][step.id] = true
            end
        end

        step_completions.each do |completion|
            topic_steps[completion.topic_id].delete(completion.step_id)
        end

        next_by_topic_id = {}
        topic_steps.each do |topic_id, unfinished_steps|
            unless unfinished_steps.empty?
                first_unfinished, t = *unfinished_steps.first
                next_by_topic_id[topic_id] = first_unfinished
            end
        end

        next_by_topic_id
    endend

There’s also a little glue code in user.rb to tie this in. I’ll spare you unless you specifically want to see it. Also, it wants the UserStepItem to have a topic_id to index on,, but that’s not hard.

Finally, ActionMailer

It’s nice to calculate these things. But how do we send mail? First, ActionMailer has very good documentation to start from.

So I’ve created a mailer called TopicReminderMailer and an action called merged_reminder. It’s “merged” because the daily, weekly and monthly all go in one email.

I added the nontrivial logic to the mailer, and Rails will cheerfully show a live preview if you build a preview action (you can see one to the right there.)

And while I was very tempted to add a lot more tracking, I’ve suppressed that urge for now. I’m pretty sure I’m procrastinating by engineering. It’s time to stop that and push something out the door.

Once we have the mailer and the calculation, it’s not hard to add a snippet to the Sidekiq job to put them together… Although it’s not quite as easy as I thought, and it took me a little debugging to get to this point:

class ReminderEmails
    include Sidekiq::Worker

    def perform
        User.all.each do |u|
            remind_steps = u.next_steps_to_remind_at_time(Time.now)
            next if remind_steps.empty?
            UserTopicItem.transaction do
                remind_steps.keys.each do |topic_id|
                    ut = UserTopicItem.where(user_id: u.id, topic_id: topic_id).first
                    ut.last_reminder = Time.now
                    ut.save!
                end
                TopicReminderMailer.with(remind_topics: remind_steps, user_id: u.id).merged_reminder.deliver
            end
        end

        ReminderEmails.perform_in(1.hour)
    end
end

This won’t last forever. Or very long. In fact, it will cheerfully page through every user on the site on every Sidekiq job. If I outgrow a few thousand users, this will get very slow… though not so slow that anybody would be able to tell by their email reminders.

In keeping with the spirit of getting something workable out the door, this does just fine. And this is one of many, many opportunities for more polish later.

Is That It?

This is pretty close to workable. It’s also a huge pain to test. I can subscribe to some daily tasks, but then I’m waiting to get emailed, and then get emailed again to make sure the reminders are working right.

(There’s a lot of debugging that I do behind the scenes. If it would make for a boring blog post, I skip writing about it. But it’s in the repository.)

Sounds like it’s time for a much-more-frequent email reminder that only shows up in the UI in development. I don’t mind it existing in production if you’d have to build your own AJAX requests to subscribe to it. I just don’t want anybody hitting the “spam me and never stop” button by accident.

And with that, and a bit more testing and debugging, I have it sending me reminders from my development server.

Isn’t Software Development Supposed to be Instant?

Is it just me or is this series pretty long? I wasn’t sure I’d be able to get everything done in one more blog post, and putting everything together seems to be taking awhile.

It often does. The “last 10%” of the work is often a big chunk of the total effort since, by nature, it contains most everything I didn’t bother to plan up-front.

And finally, after six weeks (as I write this) of effort, I have what I consider an MVP.

That always feels impossible, every time I write it, until it’s true.

Now I just need to start putting the application in places where it sees real-world use. Then I’ll know how far short this falls of “actually useful, really, in the real world.”

In other words, expect more posts about this. The rate will probably slow down from here on out, though.

Want to see more about Ruby Mad Science as it happens? You can check the rubymadscience tag on this blog, including not-yet-published ones. There’s also an RSS feed of posts at the top, or you can subscribe to my email list below. I’ll definitely email the list with these posts.

Or if you’ve bought one of my books, you get to read and discuss blog posts early with my readers as I finish them, in the #prerelease-blog-posts channel of the customer Slack.