Codefol.ioRuby and Rails Stuffhttp://codefol.io/2024-01-18T00:00:00+00:00Noah GibbsRebuilding Mechanicsburg: Gross Anatomyhttp://codefol.io/posts/rebuilding-mechanicsburg-gross-anatomy/2024-01-18T00:00:00+00:002024-03-24T12:28:47+00:00Noah Gibbs<p>I’m continuing my blog posts on Mechanicsburg, in the Girl Genius webcomic.
It will probably make more sense if you <a href="https://codefol.io/posts/rebuilding-mechanicsburg-agathas-arrival/">start from the beginning</a>.</p>
<p>It’s really obvious from <a href="https://www.girlgeniusonline.com/comic.php?date=20120201">some of the comics</a> that the authors have a strong idea of Mechanicsburg’s geography. They probably have fun sketchy maps that they’re working from. The different views of the city often have really consistent interesting landmarks. You can sometimes <a href="https://www.girlgeniusonline.com/comic.php?date=20070321">see Castle Heterodyne from one side</a> and <a href="https://www.girlgeniusonline.com/comic.php?date=20220930">sometimes see it from the opposite</a>.</p>
<p>But they don’t share those maps.</p>
<p>I get it. They like to hint at things, and to have a lot of flexibility. Reasonable.</p>
<p>But also, for a role-playing game, I need to more or less recreate it.</p>
<p>So what do we know about the major anatomy of Mechanicsburg, and where everything is inside it?</p>
<h2>Pictures at an Exhibition</h2>
<p>We get a few big views across Mechanicsburg, most prominently <a href="https://www.girlgeniusonline.com/comic.php?date=20070319">this first</a> <a href="https://www.girlgeniusonline.com/comic.php?date=20070321">two-page spread</a>.</p>
<p>You’ll occasionally see Castle Heterodyne in the background as well, and we can figure out a bit of positioning from which side we’re seeing and where we see it. For instance, <a href="https://www.girlgeniusonline.com/comic.php?date=20220930">it’s visible from the Toad Gate</a> but from roughly the opposite side from the first big two-page view.</p>
<p>In the big spread, what do we see? The Castle, obviously. The Red Cathedral is pretty prominent. The area off to the left with a windmill, smokestacks and a wall around it is <a href="https://www.girlgeniusonline.com/comic.php?date=20170908">pretty clearly the Court of Gears as described by Ivo Sharktooth</a>.</p>
<p>We have occasional other clear pictures of the area around Mechanicsburg, but not many. Most pictures don’t have a giant aerial view, because why would they? Even pictures on board airships tend to feature the characters and the airship, not the ground.</p>
<h2>Major Locations</h2>
<p>Sharktooth mentions <a href="https://www.girlgeniusonline.com/comic.php?date=20170908">the Hospital District, the Greens, the Field of Weights, the Tumbles and the Court of Gears</a>. We also know about Castle Heterodyne, the source of the River Dyne. There’s the Red Cathedral, connected to underground locations like the Crypts. Various city gates are mentioned at different times. And for now I’ll ignore places that are mentioned just once or twice (the Bone Quarter, the North Windings, the Snarling Labyrinth and many others) since they don’t help us get our large-scale geography sorted.</p>
<p>There are several gems in the text rather than the pictures that help. For instance, <a href="https://www.girlgeniusonline.com/comic.php?date=20120201">during the Siege there’s an update on troop movements</a>:</p>
<div class="highlight"><pre class="highlight plaintext"><code>Agatha: What about that group in the North Windings?
Vitriox: The Dark Light Guards? They're putting up a strong resistance, but they're falling back to the East.
Agatha: Perfect. With the Dyne to the South, they'll funnel into the Greens...
Vanamonde: Where they'll have no cover at all.
Krosp: Now you're *sure* they won't take the Cathedral?
Vanamonda: Oh, I *really* hope they try.
</code></pre></div>
<p>This tells us a lot in a really condensed way. The North Windings are going to be in the North. The Dyne is South of there somewhere — which makes sense, its source is Castle Heterodyne in the rough centre of town. Going East, they get to the Greens. And the Cathedral is nearby, so presumably in the North part of town. Now go back and look at <a href="https://www.girlgeniusonline.com/comic.php?date=20070319">that two-page</a> <a href="https://www.girlgeniusonline.com/comic.php?date=20070321">spread</a> knowing that the Cathedral is in the North.</p>
<p>It <strong><em>is</em></strong> very annoying that you can’t see the path of the Dyne in that picture. But knowing it flows from the Castle helps a bit. It may well run under the Castle, and the waterfall may only be a small part of its waters.</p>
<h2>Monsters at the Gates</h2>
<p>We also get some descriptions of where Mechanicsburg’s various gates are.</p>
<p>The big two-page spread is seen from near the Main Gate, sometimes referred to in the wiki as the River Gate, where Agatha first entered. Just outside that gate is a bridge over the big river, presumably the Dyne.</p>
<p>The Toad Gate is roughly on the opposite side of town, as judged by its view of Castle Heterodyne. We’re told it’s the nearest to the train and airship yards.</p>
<p>After Jorgi and Andre talk about landing at the Monsters Gate, the Jagermonsters come in via Mechanicsburg’s “Eastern Gates”, and so presumably the Monsters Gate is in the East. We hear about big plains East of the city, and presumably the Monsters Gate leads to those plains.</p>
<p>The Black Gate may be far from the Main Gate based on its <a href="https://www.girlgeniusonline.com/comic.php?date=20071116">signs like “Trade Entrance, Invade in Front”</a>. But it’s hard to tell if the Main Gate would be the preferred gate for invasion since it’s on a side with mountains, or if it’s easier to bring your army up to the Eastern Gates <a href="https://www.girlgeniusonline.com/comic.php?date=20071217">as did X the Destroyer</a>. But the Jagers in that comic are talking about “this” big open plain in front of the town and how armies invade. So maybe the Black Gate is quite close to the preferred invasion gate? They also refer to scenes where Agatha and Herr Diamant are watching Gil as <a href="https://girlgenius.fandom.com/wiki/Chronology_-_Volume_VII">being on the Eastern Walls of Mechanicsburg</a>, so perhaps the Black Gate is also fairly Eastern.</p>
<p>The Jagers take Gil in via the “Sneaky Gate” to Mama Gkika’s, and that one is clearly quite close to the Black Gate. So presumably in the East?</p>
<p>Several gates aren’t named. The Wulfenbach troops flee from a gate where there’s a gap in the thorn hedge, and there’s a gate where a battering ram attacks, and where Franz fights another dragon. But it’s not clear where either of these gates are, or what they’re called.</p>
<h2>Putting It All Together</h2>
<p>I’m going to have to put together a map for my campaign. So what does this all look like, when I put it together?</p>
<p><img src="Mechanicsburg-spread.webp" width="650" height="331" /></p>
<p>Based on the information above with where the Cathedral, North Windings, etc. can be found, I’m assuming the Main Gate leads into the Southwest corner of town. The Cathedral, North Windings and Greens are all up North, and the Court of Gears is visibly in the Northwest corner of town.</p>
<p>The Dyne must flow out of the Castle, then be the Southern border that prevents going South from the North Windings, instead requiring invaders to move East into the Greens. It’s not visible, but you could imagine it there, behind other visible parts of the picture.</p>
<p>Mountains surround the town to the North and West and possibly somewhat around the South. The Monsters Gate and the Black Gate are both canonically on the East side of the city. I choose to interpret the Black Gate’s “invade in front” sign as pointing to the Monsters Gate, the big entrance to the city on the Eastern side.</p>
<p>If the Main Gate is on the South end of town, and the Toad Gate is roughly opposite it in town, plus the Castle is a bit to the right when you look through the gate at it… I choose to interpret that as the Toad Gate being in the Southeast of town.</p>
<p>If you go straight in from the Main Gate, <a href="https://www.girlgeniusonline.com/comic.php?date=20070316">you’ll get to a statue of the Heterodyne Boys</a>, then turn left to get to the Great Hospital. There’s some room for interpretation there (what direction were they initially facing?) but you can see my interpretation below.</p>
<p>Which leaves questions like “where is the Field of Weights?” and “where is the Tumbles?” I feel like the East side of town is pretty barren, so I’ve put the Tumbles in the Northeast. The Main Gate and Black Gate (“trade entrance”) are both on the South side of town, and the view from the Main Gate looks pretty tourist-friendly. So I’ve put the Field of Weights on the South side of town. Something has to be, and I’d expect the Field to be fairly large.</p>
<p>Click the map below for the big version of my interpretation. And the downriver snail farms are, alas, not yet canon.</p>
<p><a href="Mechanicsburg Map 1600.png">
<img src="Mechanicsburg Map 400.png" width="400" height="300" />
</a></p>
Rebuilding Mechanicsburg: Zola's Arrivalhttp://codefol.io/posts/rebuilding-mechanicsburg-zolas-arrival/2024-01-16T00:00:00+00:002024-03-24T12:28:47+00:00Noah Gibbs<p>I’m continuing my blog posts on Mechanicsburg, in the Girl Genius webcomic.
It will probably make more sense if you <a href="https://codefol.io/posts/rebuilding-mechanicsburg-agathas-arrival/">start from the beginning</a>.</p>
<h2>A Bit More Setting</h2>
<p>A lot of the Girl Genius setting is told by implication. 200 years ago, they had
the Storm King who united Europa in peace. Then he was killed, and the Long War
raged for nearly all of that time as Sparks and noble families vied for dominance,
killing each other with armies and Mad Science. There were about 20 years of relative
peace after the Heterodyne Boys rose to power and negotiated peace in Europa,
followed by the Other’s war of destruction in which they disappeared, followed
(perhaps five years later?) by the return of Baron Wulfenbach and the establishment
of the Pax Transylvania under the Wulfenbach Empire.</p>
<p>The main story begins after around ten or fifteen years of general Wulfenbach rule. My own tabletop role-playing game story starts a bit earlier, as Wulfenbach is still becoming
established and Mechanicsburg has not yet joined him, but will soon.</p>
<p>So what’s “told by implication?” That seems obvious enough.</p>
<p>Partly, a world of constant war is a world of refugees. Mechanicsburg is historically violent, awful and xenophobic. We only know a little of the Flesh Yards (Fleshyards?) and what they did, but you can bet that a constant stream of refugees to Mechanicsburg will be involved. And so a Mechanicsburg preying on refugees during the Long War is transformed to a Mechanicsburg preying on the tourist trade during peacetime.</p>
<p>The Circle of Blood (Mechanicsburg’s crime families) are driven underground when the Boys take power, but they’re not gone. One assumes that robbing or killing tourists and refugees is comparatively acceptable. And it’s also clear that the Boys have more limited power outside Mechanicsburg proper. You’ll find Jagers hanging around small towns in <a href="https://www.girlgeniusonline.com/comic.php?date=20100519">Maxim Buys a Hat</a>, for instance, when they’re still not allowed in Mechanicsburg. It’s not just Old Man Death who isn’t surprised to see them. His daughter, his customers and the waitress all seem to think it’s perfectly reasonable for a couple of Jagers to sit and order food. This isn’t Mechanicsburg where they are forbidden, yet it’s easily close enough for them to be ready to assist Agatha. Old Man Death talks about how “young bloods from the area” used to ride with the Jagers. This is near Mechanicsburg, no question.</p>
<p>So it’s very likely that before most Jagers signed on with the Baron, that they were hanging around outside Mechanicsburg. Yet the Jagers are considered somewhat rehabilitated by the Boys. Which means they were forbidden from raiding other towns and countries.</p>
<p>Indeed it’s clear that there’s plenty of practical gray area here. The monsters and Jagers aren’t allowed in Mechanicsburg, but are fine in the basements and outlying villages. Castle Heterodyne has no trouble summoning monsters when it’s repaired, and it wouldn’t have constructed them from scratch. Carson <a href="https://www.girlgeniusonline.com/comic.php?date=20070425">is surprised to see Krosp yelling</a>, likely because he’s so obviously a construct on the streets of Mechanicsburg, yet we see the horse-construct <a href="https://www.girlgeniusonline.com/comic.php?date=20070316">leaving the city gates</a> and when the Fleshyards were destroyed <a href="https://www.girlgeniusonline.com/comic.php?date=20211011">the people of Mechanicsburg took the monsters in</a>.</p>
<p>You get the impression of a Mechanicsburg with many monsters just below the visible surface, tucked away in cellars, living deniably on the edges, able to appear as normal people or animals and so on.</p>
<p>And so in my own campaign, that’s what we have. Monsters working in businesses if they’re out of sight. Market stalls catering to constructs if the merchandise is deniable or nominally for export. Constructs used by bandit gangs, if they’re careful to rob tourists or refugees instead of locals.</p>
<p>The Heterodyne Boys demanded their monsters take an oath to leave Mechanicsburg, and that oath is binding until the next Heterodyne — Agatha — takes power. Many, but not all, of the Jagers and a few other monsters (like <a href="https://girlgenius.fandom.com/wiki/Von_Pinn">Von Pinn</a>) accepted positions with the Baron, but he’s not the one who made them promise to leave.</p>
<p>And that’s why crowd scenes when Agatha arrived had no constructs or monsters, yet <a href="https://www.girlgeniusonline.com/comic.php?date=20201130">crowds in the Solstice story</a> have monsters and constructs. And you see <a href="https://www.girlgeniusonline.com/comic.php?date=20170821">monsters and constructs on the street in Ivo Sharktooth</a>, if mostly Jagers. And of course the Monsters Guild story is stuffed with them. It’s set in the old Monster district, now the Hospital district, where the old Monster Guildhall used to be, but by then they have a restaurant <em>specifically</em> serving monsters and similar.</p>
<h2>Zola’s Arrival</h2>
<p>Let’s go back to looking at pages in order, now that we have more background.</p>
<p>When <a href="https://www.girlgeniusonline.com/comic.php?date=20070425">Zola arrives in Mechanicsburg</a>, we get some of the <a href="https://www.girlgeniusonline.com/comic.php?date=20070427">clearest crowd scenes</a> in the books. And if there are monsters or constructs there other than Krosp, they’re too subtle to pick out. No obvious monsters.</p>
<p>Also worthy of quick mention: we get a <a href="https://www.girlgeniusonline.com/comic.php?date=20070502">few random people on the castle stairs</a> and one <a href="https://www.girlgeniusonline.com/comic.php?date=20070507">distant view of the city from above</a>.</p>
<p>I’ll skip the various interior views of the Great Hospital since they tell us little or nothing about the town. And there are a couple of pages of <a href="https://www.girlgeniusonline.com/comic.php?date=20070516">talking to Carson</a> and entering the coffee shop that are mostly noteworthy for more Mechanicsburg demographics. Herr Mitrant seems French, while Rinja is likely some kind of Scandinavian and one shop employee is black, and presumably of African descent. We see a fez, traditionally Turkish, in <a href="https://www.girlgeniusonline.com/comic.php?date=20070702">more than one coffee shop scene</a> and they’re common in many Mechanicsburg scenes.</p>
<p>And post-coffee-shop they’re <a href="https://www.girlgeniusonline.com/comic.php?date=20070704">back on the streets</a> briefly and we see the Doom Bell. But after that, it’s awhile before we see Mechanicsburg again.</p>
<p>After the crypt, <a href="https://www.girlgeniusonline.com/comic.php?date=20071112">we see Mechanicsburg’s walls</a> and <a href="https://www.girlgeniusonline.com/comic.php?date=20071116">the Black Gate</a>, marked as “trade entrance” with instructions to “invade in front” on a sign pointing right. Soon we get more <a href="https://www.girlgeniusonline.com/comic.php?date=20071123">group pictures</a> with random Mechanicsburg residents, possibly its militia.</p>
<p>“In front” would presumably mean the main gate.</p>
<p>And then we skip street scenes of Mechanicsburg <a href="https://www.girlgeniusonline.com/comic.php?date=20080716">for some time</a> as Agatha enters the castle.</p>
<p>But we’ll return there in our next installment.</p>
Rebuilding Mechanicsburg: Agatha's Arrivalhttp://codefol.io/posts/rebuilding-mechanicsburg-agathas-arrival/2024-01-15T00:00:00+00:002024-03-24T12:28:47+00:00Noah Gibbs<p>My family and I have started playing
<a href="https://www.sjgames.com/gurps/books/girlgenius/">GURPS Girl Genius</a>
and I’m running a campaign that starts when Agatha is around 5
— about 13 years before Book 1 starts. I’ve been
reconstructing some combination of the modern and 15-years-earlier
setting from the
<a href="https://girlgenius.fandom.com/wiki/Girl_Genius_Wiki">Girl Genius Wiki</a>,
the comic and various other sources.</p>
<p>This is my own contribution: things I’ve noticed, things I’ve
researched, things I’ve put together and so on.</p>
<p>It’s a series of articles:</p>
<ul>
<li>This one: Rebuilding Mechanicsburg: Agatha’s Arrival</li>
<li><a href="https://codefol.io/posts/rebuilding-mechanicsburg-zolas-arrival/">Rebuilding Mechanicsburg: Zola’s Arrival</a></li>
<li><a href="https://codefol.io/posts/rebuilding-mechanicsburg-gross-anatomy/">Rebuilding Mechanicsburg: Gross Anatomy</a></li>
</ul>
<p>None of this will make a lot of sense if you haven’t read the
comics. Luckily you can
<a href="https://www.girlgeniusonline.com/comic.php?date=20021104">start right now, for free</a>
if you’re so inclined. They’re fantastic for reading out loud,
a lot like old radio shows. I read the comics to my kids and I
have since they were tiny. The Foglios have been drawing them since 2002,
and new pages show up around three a week with occasional breaks,
gaps and filler short stories.</p>
<p>Here’s some major background from the comics and Wiki that I’ll refer to often:</p>
<ul>
<li><a href="https://www.girlgeniusonline.com/comic.php?date=20170809">Ivo Sharktooth, PJ</a></li>
<li><a href="https://www.girlgeniusonline.com/comic.php?date=20201130">A Mechanicsburg Solstice Story</a></li>
<li><a href="https://www.girlgeniusonline.com/comic.php?date=20211004">The Mechanicsburg Guild of Monsters</a></li>
<li><a href="https://girlgenius.fandom.com/wiki/Internal_Chronology">Girl Genius Wiki timeline</a></li>
</ul>
<p>You may especially want to read the Ivo Sharktooth story since it contains more
concentrated geographical and social information about the city than any other
part of the comic. If I mention the Court of Gears or the Field of Weights, the
Sharktooth short story is where that background comes from.</p>
<p>I’ll also mention tidbits from the
<a href="https://girlgenius.fandom.com/wiki/Girl_Genius_Novelization">novelizations</a>.
Mostly they have the
same information as the comic, but occasionally — especially in footnotes
— they have important facts that aren’t otherwise mentioned.</p>
<h2>Some Quick Setting</h2>
<p>Mechanicsburg is in Transylvania in the books. The main Girl Genius story starts around 1892. I’ll keep mentioning 1879, because that’s where our RPG story starts.</p>
<p>Transylvania has a fragmented national character — there are a lot of
different ethnic groups there. It’s currently part of Romania, but has
historically been home to Eastern European Jews, Hungarians, Romany/Travellers,
Germans and many more. A lot of the area has been one tiny city/country or another,
changing fairly often, for centuries. Many of those countries have fantastic names
like “Wallachia.” And early on a lot of identities like “German” weren’t
really a thing — before there was Germany there were a lot of mini-countries
which (mostly) became provinces of Germany. So being a Rhinelander
or a Prussian wouldn’t necessarily involve being “German” before, what, 1866?
And there’s a whole thing with the Volksdeutsche… The relevant Wikipedia pages
are a lovely read if you have the time. I get some of this from <a href="https://www.amazon.co.uk/Paul-Kovis-Transylvanian-Cuisine-Middle-European/dp/0517556987">Paul Kovi’s Transylvanian Cuisine</a> as well.</p>
<p>While Transylvania has a lot of historical ethnic diversity, (fictitious) Mechanicsburg
has far more. The ruling Heterodyne family raided all over Europa, bringing back
a variety of people as spoils of war. There are a few ways that their
world is <em>less</em> diverse — for instance, trips to the Americas seem to be
effectively impossible (yet they have chocolate!). But mostly their Transylvania,
which prominently features African and Chinese people, is more diverse than would
have been true in our world.</p>
<h2>Some Comic-History Facts</h2>
<p>Mechanicsburg is in a weird period of its history — even more than usual. After many centuries of despotic rule by dictators, in the 1830s the Heterodyne Boys were born. They were raised by their literally-sainted mother far from the wicked influences of the family. In 1852 they started their famous adventures. Likely around the same time, their father despaired of them becoming properly wicked, and his wife Teodora fatally poisoned him to save them.</p>
<p>The Boys <a href="https://www.girlgeniusonline.com/comic.php?date=20211011">razed the infamous Fleshyards</a>
and the Monster District and rebuilt the Great Hospital in their place. The many Mechanicsburg monsters were taken in by the people of Mechanicsburg, but the Boys apparently hated them and intended to drive them out. The Monsters Guild charter was never renewed during their reign. It seems likely that the Jagermonsters’ vow to stay out of Mechanicsburg until a new Heterodyne ruled was from the <strong><em>beginning</em></strong> of their reign, not the end. And that vow seemed to apply to all their monsters, <a href="https://www.girlgeniusonline.com/comic.php?date=20111209">like Snoz</a> and the wild Jagers, not just the Jagermonsters who enlisted with the Baron.</p>
<h2>On to Mechanicsburg</h2>
<p>The first time we really see Mechanicsburg is in book 7, “Agatha Heterodyne and the Voice of the Castle”. There’s a market, presumably in the The Field of Weights, where <a href="https://www.girlgeniusonline.com/comic.php?date=20070228">they’re exchanging rumours about what happened in Sturmhalten and selling various foods</a>. There’s a money exchange in the background.</p>
<p>Given the fragmented national character of Transylvania, a money exchange is likely a big deal. It would be one of many such exchanges. Eventually the Baron’s empire brought Imperial Dollars as a standard currency through Eastern Europe (source: RPG sourcebook), but before that their currency would be as fragmented as the set of kingdoms. Through the Long War, new mini-kingdoms were rising and falling constantly. One assumes there was a strong tendency to use money that was silver or gold so that it had inherent value. Paper money would probably have been in the form of banknotes, so they were a claim of value from a specific bank.</p>
<p>In the mean time, money exchanges would be an enormous thing for a city reliant on tourism. You need the tourists to be able to pay even if they don’t have local money. That’s one reason among many that tourism took awhile to develop in our own world.</p>
<p>There are a few really interesting things about the whole market. We see black people, at least one in arguably North African traditional garb, buying and selling. There’s at least one person who looks like a construct, or otherwise Spark-modified, yelling “eggs! mostly chicken!”</p>
<p>We see very few constructs, arguably none at all, in Mechanicsburg street scenes before Agatha takes power. Beetleburg is full of them. The short stories after Agatha takes power (Sharktooth, Solstice story, Guild of Monsters) have constructs and monsters prominently on the streets and often working in the shops. But before Agatha takes power, very few. And so the green-goggled person who looks perhaps construct-like stands out, both in the market scene and in the scenes after. It seems likely that the Jagermonster vow to the Boys was part of a larger banning of constructs and monsters.</p>
<p>For the record, <a href="https://girlgenius.fandom.com/wiki/Shoutouts_to_other_webcomics">“poiled slurgs! fried onna stick!”</a> is an old joke with the Foglios, kind of like <a href="https://girlgenius.fandom.com/wiki/The_Winslow">the Winslow</a>. So I wouldn’t read any specific historical context into it.</p>
<p>The <a href="https://www.girlgeniusonline.com/comic.php?date=20070302">following page</a> has much less of note. A tour guide, and you can see a wheeled self-driven vehicle on the street below the balcony, presumably Spark-made originally. And the following page doesn’t really show the city.</p>
<p>And then we see Mechanicsburg, with Carson Von Mekkhan talking to Castle Heterodyne at the <a href="https://www.girlgeniusonline.com/comic.php?date=20070307">main gate, with its giant skulls, gargoyles and lightning devices</a>. Horse-drawn wagons go into and out of the gate.</p>
<p>The <a href="https://www.girlgeniusonline.com/comic.php?date=20070309">following page</a> gives us one of our first pieces of geography: as you look out the main gate, you can see a busy four-arched bridge below you and to your right, with (presumably) the Dyne running from your right to your left, quite vigorously. So either the Dyne is a quite large river or it joins with another, or perhaps that isn’t the Dyne. Remember that the Dyne <a href="https://www.girlgeniusonline.com/comic.php?date=20070420">flows from</a> <a href="https://girlgenius.fandom.com/wiki/Castle_Heterodyne">Castle Heterodyne</a>, but it’s hard to imagine <em>quite</em> that much water flowing from <a href="https://www.girlgeniusonline.com/comic.php?date=20070420">the waterfall we see</a> in the pictures of the castle.</p>
<p>I’m saying things like “right to left” because we don’t yet know compass directions, though I’ll assume some eventually.</p>
<p>There are also mountains in the background with a small keep on top of them. So: the big river near town flows past the mountains, and those mountains are at least somewhat settled and defended.</p>
<p>The <a href="https://www.girlgeniusonline.com/comic.php?date=20070316">next page</a> has one <em>really</em> interesting thing from my point of view. There is a wagon drawn by a horse-headed construct in the first frame leaving Mechanicsburg. Presumably the wagon and construct came from inside. Again, we see nearly <strong><em>no</em></strong> constructs in Mechanicsburg during this time. Krosp is covered in bandages, but it’s not clear to me why. Later they let him prance around in Mechanicsburg uncovered, talking and wearing his coat, so presumably it’s not a problem for him to be recognized as a construct or as a specific distinctive cat-construct that was travelling with Agatha.</p>
<p>Next let’s talk about the <a href="https://www.girlgeniusonline.com/comic.php?date=20070319">two-page</a> <a href="https://www.girlgeniusonline.com/comic.php?date=20070321">spread</a> looking over Mechanicsburg that follows. Most of the signs are references to other webcomics (e.g. Wapsi Square, Girls with Slingshots, Dominic Deegan.) But there are several interesting things about the buildings, too. We see several Islamic- or Russian-style onion domes, including one on the Cathedral and one on Castle Heterodyne. There is also a building with a stair-stepped facade in (to my eye) a similar style, possibly like Moorish Spain. This suggests a general Islamic influence on the architecture, not limited to Mosques or similar specific buildings. This would be in line with the general cosmopolitan feel. Like Glasgow, Mechanicsburg seems to be a city with very diverse architectural influences, even aside from buildings that seem intentionally Mad-Science-y.</p>
<p>Castle Heterodyne is quite centrally located, but has <a href="https://www.girlgeniusonline.com/comic.php?date=20070416">an outer wall around it that we’ll see more closely later</a>. So it’s very much in the center of the city, but also a bit set apart from it. It may be the “fortress of the Heterodynes”, but you don’t get the impression that it is frequently directly assaulted. The outer layers of Mechanicsburg, including mountains and outlying towns, are already a legendary defense. You’d have to get very far indeed to assault the castle itself.</p>
<p>On the left page is a large industrial and factory district full of smoky chimneys, either outside the city walls or surrounded by a separate wall inside the city. This could arguably be the Court of Gears, where Sparky and industrial products are sold, and apparently well-known for its stink. But it’s hard to be sure, since Mechanicsburg is likely to have more than one district of that type. Also: no obvious signs of gear logos or similar. (Edit to add: yeah, that’s probably the Court of Gears. In <a href="https://www.girlgeniusonline.com/comic.php?date=20170908">Ivo Sharktooth when he describes it</a> you can see the windmill and outer wall.)</p>
<p>It’s hard to tell if Agatha and company head into the city where they were told: down the boulevard, to the state of the Heterodyne boys, then left toward the Great Hospital. They’re riding past souvenir stands, but those will probably be constant near the main gate, near the Great Hospital and generally where tourists are expected.</p>
<p>Lots of tourists near the Great Hospital? Yup. Von Mekkhan absolutely expects that they’re bringing a sick child for treatment, and the Great Hospital is basically peerless in Europa for its quality of treatment. They live in a world of refugees, which has been rocked by war and Spark-created disasters for the vast majority of the last 200+ years. You would expect a <strong><em>ton</em></strong> of visitors to the hospital.</p>
<p>While the <a href="https://www.girlgeniusonline.com/comic.php?date=20070328">interior areas of the hospital grounds aren’t full of tourist advertisements</a>, you’d assume that all the nearby areas of the city are.</p>
<h2>So Then… What?</h2>
<p>This is a good start. We have some history, some observations, and a bare beginning of some geography. There are a lot more comic pages that show Mechanicsburg, often with wonderful little Easter eggs hidden here and there.</p>
<p>And of course, I’m being gamemaster for a tabletop role-playing game. These little cultural bits and trivia are <strong><em>exactly</em></strong> what I’m here for. I’m just writing them down for you too.</p>
<p>This article links every page of the comic that shows Mechanicsburg city scenes before Zola’s arrival. So I’ll probably pick up <a href="https://www.girlgeniusonline.com/comic.php?date=20070425">there</a> next time I write. There’s a lot to be said about those crowd scenes… starting with the notable lack of monsters and constructs.</p>
Caffeine Managementhttp://codefol.io/posts/caffeine-management/2023-10-02T00:00:00+00:002024-03-24T12:28:47+00:00Noah Gibbs<p>I have a pretty regimented routine of <a href="#">powdered caffeine usage</a>. A lovely fellow I know asked me about it recently. Occasionally people do. And I thought, I should just tell everybody.</p>
<p>To do that, I’ll first have to summarise something that was once online and I can’t find these days: agonistic versus antagonistic usage of stimulants. It’s simpler than it sounds.</p>
<p>And I’ll describe the eight-week cycle I’m on to manage my tolerance. Caffeine is an addictive drug that builds high tolerance. Most people ignore that, build their tolerance and then don’t get much benefit from it afterward. That problem can be fixed.</p>
<p></p>
<h2>Agonistic Vs Antagonistic</h2>
<p>The short version goes: you can take caffeine when you’re tired at night to stay up longer. You won’t feel as tired, you’ll be able to focus better than when you’re exhausted, the reduced sleep will be mostly okay until you can catch up later.</p>
<p>This is called “antagonistic” use of a stimulant. Your body is trying to do something (go to sleep) and you’re working against it. The caffeine lets you have more hours of nearly-as-good-as-well-rested time when usually you’d have exhausted time.</p>
<p>You can also take caffeine bright and early in the morning, to help yourself focus better than you usually would. It can make you more energetic, and to some extent smarter, than you otherwise would be. By the time you’d be going to bed, the caffeine is wearing off. So you’re smarter during the day, but you don’t normally get an extended amount of daytime, just extra energy and focus during normal waking hours.</p>
<p>This is called “agonistic” usage, the opposite of antagonistic. Your body is doing something (being awake in the morning) and you’re increasing or exaggerating it.</p>
<p>Stimulants take a toll on your body. “Coffee is the theft of time, which by rights belongs to your future self” says Terry Pratchett. That’s roughly true. You can move your energy and focus around to when it’s most valuable to you. But you can’t get significantly more. One way or another, you’ll need to pay the cost of it.</p>
<p>There’s a natural curve to your attention and focus. You can’t just burn energy all the time. If you use a stimulant agonistically, you’re trying to increase that curve – more focus when you’re focused, and even less focus during your down-times. If you use it antagonistically, you’re trying to flatten that curve – extra focus during down-times, at the cost of being more tired during your higher-focus times.</p>
<p>You shouldn’t mix agonistic and antagonistic usage. I’d recommend separating them by at least a few days if you can. Coffee can make you brighter and perkier. It can make you stay up late at night. But if you try to combine the two, you’ll normally have a bad time. The increased body load is real. You’re also messing up those curves. You can’t tell your body “you don’t get rest” for very long. At most, you can change <strong><em>when</em></strong> your body takes its rest, and how much you tire yourself out by hard focus and vibrant energy.</p>
<h2>My Daily Routine</h2>
<p>I’m an agonistic caffeine user. I’m a computer programmer, and mostly I need to be smarter than I could be naturally. If that’s only for a limited number of hours per day, I’m fine with that. My un-aided number of hours of enhanced intelligence is zero. If I can bring that up to, say, three, that’s a huge benefit to me.</p>
<p>You could do the same thing with antagonistic usage – have quieter, longer days at lower focus. I can think of jobs where that’s useful, or it might just be a thing you want for yourself. I assume it would work very similarly. But it’s not something I’ve done for myself, so I won’t tell you about it.</p>
<p>Since I’m an agonistic user I have my caffeine early in the morning, usually around 300mg. I mix it into tea or coffee since caffeine is really bitter. Putting it in milk or orange juice, for instance, tastes terrible.</p>
<p>I’m careful to avoid having caffeine later in the day. Remember: you shouldn’t mix agonistic and antagonistic, so you have to be somewhat disciplined about that. It’s not a 100% always rule, but having a lot of caffeine later in the day <strong><em>will</em></strong> throw off my rhythm for a few days.</p>
<p>I tend to have caffeine Monday through Friday and then skip Saturday/Sunday. Not only do I do professional work M-F, on Saturday and Sunday I’m often doing things with my wife and kids, not sitting and programming. The extra no-caffeine time makes it easier to catch up on sleep and not build tolerance too fast.</p>
<h2>Tolerance Resets</h2>
<p>There’s a problem with the obvious “just do that daily” method. You build up significant tolerance to caffeine, and eventually it’s a lot less effective at making you smarter. There’s a related problem when you’re too tired, whether from hard labour or skipped sleep. If your body doesn’t have the extra “oomph,” the caffeine will mostly feel like a non-event.</p>
<p>In both cases, the first thing to do is to stop taking caffeine for a bit. Tolerance builds regularly, so I have a relatively simple pattern of six weeks on and 1.5-2 weeks off. I’m going to a work event in Amsterdam as I write this. It throws off my schedule, so I’m extending the caffeine break to more like 2.5 weeks. But I try for minimum 1.5 weeks. I don’t avoid caffeine 100%. I usually have a cup of Earl Grey in the morning. But compared to my normal, that’s still a great way to reset tolerance.</p>
<p>I’ve tried shorter reset breaks. Some sources will claim your body can reset in at little as 2-4 days. My experience has been that to reset both physical tolerance and “body habits,” it really does take over a week. Sometimes 1.5 weeks is enough for me, sometimes it isn’t, so I try to make it to 2 weeks. That also makes for a calendar-friendly cadence of six weeks on, two weeks off.</p>
<p>I’ve tried resets without even the tea. For me it hasn’t been any more helpful. And I’ve tried breaks of a month or more. That <em>is</em> slightly more helpful, but not enough, for me, to be worth the extra reset time.</p>
<p>I used to get caffeine headaches when doing the no-caffeine weeks. I still get the occasional fairly mild one. But mostly the combination of a short – six weeks – time building tolerance with practice and repetition has been enough to get rid of them.</p>
<p>If it seems like your tolerance is getting high, also make sure you’re getting enough sleep. You won’t stay away significantly more hours this way, but you may have a higher-intensity routine anyway. Sometimes you just need more sleep to make up for more focus. The “hey, nothing happened” feeling of high tolerance is hard for me to tell apart from the “hey, nothing happened” when I’m just too tired.</p>
<h2>Doctors and Medicating</h2>
<p>Some folks with ADHD in the audience have been doing some mental math. ADHD is commonly treated with very caffeine-like stimulants (e.g. Ritalin, Adderall). It’s not unknown for ADHD folks to self-medicate in this way.</p>
<p>Caffeine is quite easy to get compared to prescription-only ADHD medications. It’s also quite cheap. $20 or so of powdered caffeine can last multiple years, and it may well expire before you run out.</p>
<p>If you can get a diagnosis it’s probably worth getting the real stuff. Caffeine builds tolerance much faster than prescription ADHD medication, which requires much stronger discipline in how you take it – including during times when you’re off your meds to reset your tolerance. The prescription medication <strong><em>is</em></strong> just better.</p>
<p>That “if you can get” last paragraph is doing a lot of work. Many folks have a hard time dealing with doctors for all sorts of good reasons. If that’s you, or if you have a long delay before you can see a doctor, I’d say go for caffeine. You’ll know more about how you respond to it by the time you can get a diagnosis. The presciption medication will still be a step up.</p>
<p>You may well want to stop taking caffeine before you actually see the doctor, though. A distressing number of doctors treat “I have this mostly under control with a Rube Goldberg machine of bad coping methods” as being the same thing as “fine, really”. And they often respond very poorly to you explaining complicated situations or treatments. Instead, just have a reset period (again, around 1.5-2 weeks) before you see them and let them diagnose you while untreated. If no good comes of it, you can always start back on your normal caffeine cycle. You and I both know that we’re talking about intense use of an addictive stimulant. But like alcohol, it has many years of precedent and we pretend that makes it fine.</p>
Status Pages, and Why Companies Can't Be Relied Onhttp://codefol.io/posts/status-pages-and-why-companies-cant-be-relied-on/2023-09-29T00:00:00+00:002024-03-24T12:28:47+00:00Noah Gibbs<p>In the age of using internet sites for important things – communication, say, or banking – we’ve grown status pages, to let companies know whether their service is currently working.</p>
<p>In fact, a lot of companies started growing <strong><em>external</em></strong> status pages, so <strong><em>other</em></strong> people could tell you whether their site was working. That’s a lot of what let status pages happen at all. If you don’t provide it, somebody else will.</p>
<p>Why is it so bad for somebody else to provide that? What’s different about a company’s official status page?</p>
<p>As I write this there’s an ugly Discord outage which is barely acknowledged on their status page, so it’s a great time for me to talk about that.</p>
<p></p>
<h2>Experts Agree, Everything is Fine</h2>
<p>Here’s a weird thing: company status pages tend to far downplay an outage. If people are showing up because they think something is wrong, what advantage do you get from lying to them? For instance, here’s what Discord is showing as of a moment ago:</p>
<figure class="">
<a class='lightboxed' href='discord_no_api_errors.png' >
<img src="discord_no_api_errors.png" class="" alt="a status page showing a recent incident with API error rate, and below it a graph showing no problems, now or past, with APIs" />
</a>
</figure>
<p>Notice anything odd? They’re showing an incident with API error rates, but show no changes in the API uptime. So then “error rates are bad, people can’t use it” doesn’t count as affecting the uptime.</p>
<p>I’m not trying to pick on Discord here. If you look at nearly any company during an outage they do this. They tell you everything is fine, and has always been fine, even mid-outage. They also put up the “we found a problem” notice long after the problem is found – usually after an article has been published and passed around on social media – and put up a “we solved it, everything is fine” message before it’s solved. Here’s Discord’s “everything is fine” notice, from lower on the exact same page at the exact same time, while at the top it (correctly) says people are having problems:</p>
<figure class="">
<a class='lightboxed' href='discord_api_response_time.png' >
<img src="discord_api_response_time.png" class="" alt="a status page showing the API error incident as being fully resolved 20 minutes after it started, while it's still ongoing" />
</a>
</figure>
<p>As I wrote this they (finally) put up something acknowledging (some of the) widespread problems with this new update, in a similar downplaying tone.</p>
<p>Here’s the thing: people go to this page because they already know Discord is screwing up. They can instantly communicate with a whole world of people to verify that, yes, Discord is screwing up. Third-party status pages like <a href="https://downforeveryoneorjustme.com/">DownForEveryone</a> say that, yes, Discord is screwing up.</p>
<p>Is Discord that oblivious to problems, such that external status pages are <strong><em>that far</em></strong> ahead of their own assessment? And so is every company with a status page?</p>
<p>Of course not.</p>
<h2>Down For Everyone Or Just Me?</h2>
<p>The first popular status pages, things like <a href="https://downforeveryoneorjustme.com/">Down For Everyone Or Just Me?</a> did what that sounds like. You want to know if the problem is just you. It’s still decent, and things like it often work well. There’s not a ton of ways to make money providing this service, so mostly you have to rely on the company for this kind of information.</p>
<p>SREs are the other big cause of status pages. If you’re dealing with a vendor, and you’re reporting problems, it can be frustrating that they don’t respond to you, saying yes, they know about the problem and are working on it. And of course they’re not going to give you regular updates – why would they? You’re not their boss. And saying to a customer “oh yeah, we’re not working right now” is something the company can be sued over. It’s much safer to not say anything. In fact, most companies will tell their employees absolutely not to say anything like that, and instead to direct queries to the PR team, media team or similar.</p>
<p>A status page is a way around that, somewhat. If your operations folks can put up a status page, that updates customers, it tells everybody, and it doesn’t require extra communication during a crisis. You can just leave the status page up and everything should be fine.</p>
<p>Ops folks, in my experience, are famous for being tactless, direct and pessimistic. Their internal dashboards are often accurate and unadorned. They pride themselves on knowing when something isn’t working. Their history graphs are full of blips and ups and downs, showing when there are minor delays, or normal failovers.</p>
<p>Nothing like a public status page, in other words.</p>
<p>So what’s going wrong? Why is the public information so bad? Why is the only permissible status page a wall of green for all time, even during outages?</p>
<h2>Liability and Public Relations</h2>
<p>I’m checking the status page again now, a few minutes later, and the problem with IP addresses being widely blocked has been removed as though it never existed. It’s a real and current problem, so they’ve replaced it with the “all fixed” problem with API errors, which doesn’t show on their graphs and was fully corrected hours ago, but has somehow miraculously re-opened, but only on one part of the page, not the other.</p>
<p>It’s still down, of course. Has been for hours. This is not a case of slightly different points of view. By any reasonable metric they’re outright lying. The various other third-party downtime detectors are all well aware that they’re having problems, and Discord clearly does as well.</p>
<p>Here’s the problem: a status page, being public, becomes a weapon of public relations. The company wants to convince you that they’re reliable. They believe that a graph, from them, acknowledging problems, is less okay than them lying constantly through every outage and error.</p>
<p>Presumably they’re right. If every company does it, that’s normally a sign that it’s the right – or at least profitable – thing to do. A “best practice,” as they say in the business.</p>
<p>You may be certain, looking at these pages, that the PR teams have won over the operations teams when it comes to informing the public about problems.</p>
<p>Here’s an interesting piece of fallout from that: you can’t trust any of these services to be reliable.</p>
<p>(“These services?” At a minimum, I mean any company with an all-green status page history. But really it’s nearly any tech company. I could mince words about some companies publishing some percentage of the postmortem reports that they promise, but I won’t bother.)</p>
<p>You can’t trust them to be reliable, because they clearly believe that simply lying to you about being reliable, and pointing you at the page full of lies, is sufficient to project the image of reliability. And again, this is a nearly-universal practice, so the <a href="https://en.wikipedia.org/wiki/Occam%27s_razor">Occam’s Razor</a> assumption is that they’re right.</p>
<h2>But the Efficient Market Hypothesis?</h2>
<p>You might reasonably say, “how could a whole industry lie about this? Wouldn’t they get caught?” I tend to think of Kyle Kingsbury, a.k.a. Aphyr, and his many-year exposure of how bad the distributed storage industry is. <a href="https://danluu.com/why-benchmark/">Dan Luu has a great writeup here</a> if you want the details. I was paying attention as it happened – many of us were – and while we cheered from the sidelines, it was obvious to many people before him how bad most of those products were.</p>
<p>Fundamentally, in cases where the company has all the incentives and it’s hard for a consumer to be sure about things, the company will lie just exactly as much as they can get away with. And that’s a lot. If it’s hard to disprove them, and there’s very little penalty for lying, why wouldn’t they?</p>
<p>The problem with, say, putting up your internal operations metrics for how well things are doing is that it’s worrying. You look much more reliable if you just put up a standard wall-of-green status page.</p>
<p>Which means that reliability is treated as only a problem to the extent non-expert, constantly-lied-to customers catch you at it.</p>
<p>If you ever need more reliability than that… mostly you’re out of luck.</p>
<p>But surely you can buy special, more-reliable products from companies where that matters? In theory, yes. For instance, you could imagine buying distributed storage products from companies whose entire business is providing that safety at scale. And Aphyr reminds us that if there aren’t third parties keeping them honest, mostly they will produce complete garbage, if you actually need reliability.</p>
<p>Lying is cheaper than telling the truth. The <a href="https://en.wikipedia.org/wiki/Efficient-market_hypothesis">Efficient Market Hypothesis</a> tells us that, if nobody is keeping them honest, companies that bother to really do the work should be out-competed by companies that take the cheaper “lie on our status page” approach.</p>
<p>An efficient market or company will be efficient at following its incentives, not necessarily at doing something you want.</p>
<figure class="">
<a class='lightboxed' href='discord_okay_small_problem.png' >
<img src="discord_okay_small_problem.png" class="" alt="the status page shows that, okay, some users are temporarily blocked, while still not showing any useful acknowledgement that things are down" />
</a>
</figure>
<h2>But Why Do We Care?</h2>
<p>You can easily say, “yeah, sure, companies talking in corporate spin, news at 11.” Which is reasonable, certainly.</p>
<p>But here’s the thing: your <a href="https://codefol.io/posts/your-value-proposition-as-a-software-developer/">value as a software developer</a> depends on <a href="https://codefol.io/posts/why-keeping-software-working-is-a-superpower/">keeping things working</a>. If companies – if vendors – can’t be relied on to do that because their incentives put them in an ugly spot, that’s important for you to know. Yes, companies talking in spin, news at 11. Yes, companies being far less reliable than advertised, and lying to you about it, news at 11.</p>
<p>But that means you need to plan as though that were true.</p>
<p>Since your value depends on keeping things working, and keeping things working is harder than advertised, it’s important for you to think through that. How can you do more for yourself? How can you do more with components you can validate for yourself?</p>
<p>I plan to keep writing about keeping things working. There’s a lot to think through.</p>
<p>And here’s an important one: you can’t trust companies, or even industries, to tell the truth here. The Efficient Market Hypothesis is not coming to save you. You’re going to have to actually think through this for yourself.</p>
<p>Because otherwise, your stuff doesn’t keep working. And the value you promised doesn’t materialise.</p>
<p>I don’t want to be in the business of fooling suckers into paying me more than I’m worth. I want to be in the business of selling value. And in a world like this, that takes understanding. Let’s keep looking for understanding together.</p>
<p>As I finish writing this, they’re back to acknowledging the other problem (see above.) All the bars still show 100% reliability for the last 90 days, of course. Technically push notifications shows only 99.99% uptime. I’m not sure what was so bad on that one, down near the bottom of the graphs, that it was allowed to show imperfection.</p>
<p>Am I picking on Discord? Not really. I’m taking out my frustration on our entire industry as they “just follow best practices.” They’re not worse than the rest. And that’s the problem.</p>
Ways to Keep Software Workinghttp://codefol.io/posts/ways-to-keep-software-working/2023-09-27T00:00:00+00:002024-03-24T12:28:47+00:00Noah Gibbs<p>I can talk about <a href="/posts/why-keeping-software-working-is-a-superpower/">keeping software working</a> and <a href="/posts/mission-driven-companies-and-what-lets-a-company-live-long/">why we care</a> and theorise all I like. But normally people have something decent working on the ground long before the theorists catch up. What’s working on the ground? How do we currently keep software working?</p>
<p>I’m writing this with an eye toward individual software developers keeping things working by themselves, which says a few things about methods and budget. So let’s look!</p>
<p>We use many methods, with many tradeoffs. And we all use a mixture of them.</p>
<p></p>
<h2>Manual Testing</h2>
<p>Manual testing is very expensive, and very effective. We use it constantly when developing a feature, and a lot near software releases. A human can find problems that no automated test is likely to.</p>
<p>I think the freeform nature of human testing is underrated. We constantly look at software with thousands of automated tests, all currently passing, and say “oh, yeah, that’s completely wrong.”</p>
<p>Manual testing is especially prone to ugly “works on my machine” bugs. Not only is it often done on the original developer’s machine, it’s nearly always done on a small set of machines that are rarely reinstalled. And if you go the other route and reinstall them constantly (or use Docker or emulation) then you have the opposite problem – you don’t see the cruft of real-world machines with real-world user programs running on them.</p>
<p>Manual testing tends to fare poorly in the “one guy keeping stuff running” arena. It’s a ton of ongoing work, often for projects that have limited ongoing value.</p>
<p>There are a lot of ways to foist more manual testing off on the final user - report exceptions when they come up, push immature first versions and wait for user complaints and so on. None of that is free. It requires people on the developer side, often customer service employees, to listen to user feedback and aggregate it. Even reported exceptions needs somebody to check the exception logs. But if you’re a large company with a lot of employees, the savings in manual testing can be worth the tradeoff of pushing it onto your users.</p>
<h2>“Don’t Touch That”</h2>
<p>A lot of keeping things working is to make the system as unchanging – as rigid – as possible. This isn’t a perfect method, but it has a pretty amazing track record. Think of that old Linux server that’s been running in a closet somewhere for 25 years without a reboot. Or software written for Apple II computers. It’s still emulatable, but also if you can find an Apple II where the hardware hasn’t died of age, it all still runs fine.</p>
<p>Traditional, pre-SRE operations groups at companies were/are masterful at this. If they use an operating system, they lock updates so they can examine them coming in, one at a time. If they use a piece of software, restrict it to particular versions.</p>
<p>Many people are yelling at the screen “but security!” here, and they’re not wrong. One reason that no modern operating system allows this by default is that we don’t want massive numbers of exploitable old computers running ancient software and being used to attack the rest of us (just like with the Internet of Things.)</p>
<p>Nevertheless, this is overall a very good method. It <em>does</em> rely on restricting functionality a lot, which those operations groups were also quite good at.</p>
<p>“Restricting functionality?” Yeah. If the old Linux server in the closet was set up <strong><em>well</em></strong> 25 years ago, the sysadmin shut down all services that weren’t vital to its operation. It wasn’t a fun toy with a shiny-as-of-25-years-ago graphical interface. It ran SSH to allow maintenance and had three other open ports, all for the services it was actually used for. And then, magically, you can ignore 25 years of security bugs for all the services it doesn’t actually have installed or running.</p>
<p>An additional form of restricted functionality is that you tend to get new software slowly. If it’s locked down to allow auditing, your new versions arrive at the speed of human auditing.</p>
<p>This is one of the most effective methods for solo developers, if they’re disciplined. The long-term cost is extremely low. But of course, “restricted functionality” may defeat the whole point, depending what you want.</p>
<h2>Testing and CI</h2>
<p>For this section I’m thinking of developer-style testing: unit testing, integration testing and their numerous relatives.</p>
<p>Testing can be a huge component of keeping things working in the face of change. And yet tests, by themselves, don’t fix anything.</p>
<p>CI systems – systems that keep your tests running more or less constantly – have become far more common in recent times to let us know when something is broken. That way when it breaks you’ll get an email (or other notification) pretty rapidly, and a human can get to the business of fixing whatever is wrong.</p>
<p>Tests are a fantastic way to let your system make approximate guarantees (and those are the only kind of guarantees in the real world) in the face of change. Either an assumption will stay true, or a human will be told that it isn’t and they can work to fix it.</p>
<p>Tests are especially valuable when something big may change. That’s why they’re so good when you’re about to do a big refactor. It’s also why they’re used around other people’s code, APIs and so on. Even if you don’t change anything about your software, the rest of the world keeps changing.</p>
<p>As a solo developer, tests can be very useful. They help reduce the cost of change significantly, and often do the same for the initial development as well. The trick is to do careful, directed testing of things that may actually change. It’s easy to over-test – to write many tests for conditions that will never change, and therefore spend a huge amount of time running tests that will never fail. A test that absolutely <strong><em>will</em></strong> never fail, or will only fail when another test fails is giving you no information.</p>
<h2>Production Monitoring</h2>
<p>The flip side of developer testing is watching the real software in its final location and seeing how it’s doing. Production monitoring can be passive, watching software as it operates and looking for signs of distress. It can also be active, where you add load or make API requests and see how the system responds. It’s very common to use both.</p>
<p>Production monitoring, like the various sorts of testing, can only tell you when things break. It’s a method to manage change, not to reduce the amount of it.</p>
<p>I mentioned earlier that you can reduce manual testing by having users do more of that testing for you. Production monitoring is how you make that bet pay off, by noticing when something goes wrong for the user. Making effectively-perfect software is so expensive that nobody can really afford to do it. Production monitoring helps bandage the wounds of releasing imperfect software into the world, by noticing where its imperfections matter and where they don’t.</p>
<p>There are many variations of this, depending on whether you monitor servers, clients, standalone applications or something else. But for all of them the basic idea is the same: gather data on how your software is working, collect the data, and get it into a usable form.</p>
<p>The best production monitoring data can be used in a freeform way when humans spot problems. This is similar to the divide between manual testing and developer automated tests: specific automated testing catches known bugs excellently and cheaply, but won’t normally catch unexpected problems. You can collect general-purpose data such as CPU usage and API usage (e.g. calls/minute) to help a human verify what’s happening when problems pop up.</p>
<h2>Buy It, Make it Commodity</h2>
<p>Like “Don’t Touch That,” above, this is a way to try to reduce the amount of change rather than to cope more gracefully when change <strong><em>does</em></strong> occur. It’s massively valuable because it’s a way to <strong><em>do less</em></strong>, the most powerful technique of all.</p>
<p>The best “make it commodity” approaches use interfaces that are simple and will be around forever. If you can build a site as static HTML with no dynamic Javascript or backend, it’s hard to imagine a situation where you can’t keep it running. If you need Javascript, that’s still far better than needing a server to be live – provided the Javascript can be hosted completely on your page, rather than using a vendor API to do your storage for you (and yes, I use one of those for the comments here).</p>
<p>This is related to approaches where you use a common file format, such as writing in markdown, with the assumption that you’ll be able to convert it later. It <em>is</em> very likely that you’ll be able to turn markdown into something useful, basically forever. This also lets you change out one vendor or piece of software for another, such as switching from Netlify to Render for hosting your static HTML. The format is identical and the switch is quite painless.</p>
<p>In the long run, vendors hate this and try hard to work around it. The last thing they want is actual competition or the ability for you to take your data elsewhere. So you <em>will</em> be playing a game where vendors sell you on the common format and then try to get you to make vendor-specific extensions in any way they can. That’s part of the game and I won’t tell you how to respond, but keep it in mind.</p>
<h2>Throw People and/or Money at the Problem</h2>
<p>I’ve been mostly talking about solo practitioners and how they handle this. But it’s worth contrasting with the normal corporate approach – throwing labour at the problem continuously. This includes, but is not limited to, the manual testing above.</p>
<p>This is the fallback when any of the optimisation methods above fail. Not enough automated testing? Do it manually. Not enough automated production monitoring? Do it manually. Not enough manual testing or monitoring in advance? Hear from the customers and go check/fix it manually. Didn’t use a commodity interface? Fix up what you have for a new vendor if you switch. No interest in locking down versions? Test each new version as you switch to it, and constantly check whether the OS vendor shipped something recently.</p>
<p>It’s not that any of the activities above must be followed perfectly. It’s that the consequence of <em>not</em> following them is to spend time, labour and/or money doing it this way.</p>
<p>Why do I call this the “normal corporate approach?” Surely, companies use the methods above (it’s true, they do!).</p>
<p>A company is normally “pragmatic” in the very specific sense that they don’t like finicky or error-prone methods that require unusual competence. A company is in the business of doing something repeatedly, with replaceable parts and replaceable people. They are basically always on a marketing treadmill of constantly adding new features, often unexpected new features that weren’t planned for. All of this results in limited usage, or inconsistent usage, of the methods mentioned here.</p>
<p>You can “do less” until a manager says, “hey, we’ve added these three features and they all need to ship this year.” You can “not touch that” until the new feature needs a lot of updated packages you haven’t had time to test properly. You can use a commodity interface until you’re told to switch vendors, or add features that the commodity interface doesn’t easily support – if there’s a quicker way to do it by adding a vendor-specific API, a company is very willing to add the vendor-specific API. Companies are absolutely infamous for “driving without changing the oil” by requiring lots of new features and refusing to make them lower maintenance or add enough automated tests for them.</p>
<p>All of this can be done, provided you’re willing to make up the shortfall in extra labour. If a company makes enough money to cover the salaries, that can work great. If you’re not hiring people or bringing in revenue with features, you may need to consider whether this method will work well for you. For many of us, it’s a non-starter.</p>
<p>Which means you have to be careful about emulating companies. The results they get may not be results you want.</p>
<h2>Do Less</h2>
<p>I mentioned this briefly, but it has a lot of sub-methods. The one above, using commodity interfaces and replaceable vendors, is just one of many.</p>
<p>You can literally cut scope - removing features you might want to limit the cost of maintaining them. Doing this is an art as much as a science. How can you get the closest to what you want with the least code and the fewest features? But practice helps, and you should definitely be practicing.</p>
<p>You can create a sort of “drop box” for work. When you hit an unusual or exceptional case, have it literally email you and you can take care of it manually. If the case is hit constantly this would be a lot more work. If the case is never hit, this would be a lot <strong><em>less</em></strong> work. It’s one of the more literal interpretations of the <a href="https://en.wikipedia.org/wiki/You_aren%27t_gonna_need_it">YAGNI</a> (“You Ain’t Gonna Need It”) principle.</p>
<p>There is an old Operations trick of asking, “can you do that with an interface we already supply?” Sometimes you can limit the later effects of change by relying on something you already rely on, even if it takes a little more up-front work. Expose yourself to less change. It is, in some sense, a way of doing less – don’t add that new interface, and thus don’t add the additional labour it will entail over time.</p>
<p>In all of these cases, you’re making a bit of a bet. The email dropbox is an excellent example: it can be more or less work, and you can wind up doing more or doing less, by making that bet. If you do it with something that happens a lot, you’ve added the email dropbox for nothing. You’re just going to need to write the real version anyway. But if you do it for a feature that nobody ever uses, it’s pure win.</p>
<p>The trick with doing less is to understand that, as people who write software, we’re inclined to believe that people will use our stuff. We believe that far more than has historically been true. We’re usually wrong, at least partly. And so by building less, we’re acknowledging what the universe has been telling us the whole time: we are much more enthused about the features we write than random people are.</p>
<h2>But What’s the Upshot?</h2>
<p>I’ve talked about a number of techniques to help keep software working with limited labour and maintenance, and some common reasons that we usually don’t use them routinely, and wind up needing a lot more labour and maintenance.</p>
<p>There’s a lot going on above. Some of those are valuable techniques in isolation, while others need to be combined. How do you put it all together?</p>
<p>I’m planning to write about feedback loops and the basic mechanism of tying it all together pretty soon. But this article is already long enough.</p>