Tues Jun 20 2023

Wildcat Website: Nose-to-Toes, Part Zero

by e(Liz)abeth (Mars)ton

As I promised in the first post, I’m doing a seven-part series on how and why I made this site as I have, cobbled together from teeny tiny pieces of Internet.

At any point along the way, I could have used one of the dozens (hundreds?) of existing ‘Content Management Systems’. There are any number of great options out there, and a panoply of beautiful templates. And there are an even greater number of easy hide-the-details hosting options, such as Vercel, Netlify, etc.

But instead, I opted to start with a bare OpenBSD instance and build up from that.

This culminated in building my own markdown-friendly blog framework atop the latest bleeding-edge version of Next.js; for most of the first two months of the project, I was building off nightlies(!), as I was using features that had yet to be released. I wound up creating my own blogging framework, still very much a work-in-progress.

In this first segment, I’m going to talk about the rationale for such a radical approach, and the benefits (and costs!) I’ve accrued in the process.

While I’m extraordinarily proud of the outcome, the road I took to get here is not for everyone. But the investment in myself and in my code is already paying off.

Since the point of a ‘wildcat website’, as I’m defining it, is to start with the hosting and move up the stack from there, I’ll begin by thanking the amazing folks at OpenBSD Amsterdam for their fantastic product.

OpenBSD, in case you haven’t heard of it before, is the reason why I refer to myself as the second most security-conscious entity to emerge from Alberta, Canada. It’s a full-featured UNIX-alike with a twist: whereas other projects have concentrated on ease-of-use, containerization, or idempotence, OpenBSD has instead chosen to focus like a laser beam on security, and has maintained that focus for two decades and counting.

OpenBSD’s development culture systematically prioritizes security, and one of the most important mechanisms that provides security in OpenBSD is the extensive code review that the base system undergoes. Every line is scrutinized for vulnerability. The result is that OpenBSD has gone for ages and ages without a major CVE.

Its security is so good that other operating systems now depend on spun-off parts of the OpenBSD project to defend their most vulnerable entry-points.

Notably, OpenSSH, originally a sub-project of OpenBSD, is the de-facto standard for secure remote administration. It is no exaggeration to say that AWS, Azure, and Google are built atop OpenSSH as much as they are the Linux Kernel and Git.

Finally, one of the most important features of OpenBSD has been a side-effect of their code review policy: an underappreciated property I am calling churn resistance. For whereas a typical Linux distro now looks nothing like a Linux distribution from 1999, OpenBSD has remained markedly consistent in both its userland environment and core API.

This means that, as an engineer, OpenBSD skills you learn last much longer than equivalent Linux (or Windows, or MacOS) skills. This has important benefits for security, as it means that you can reason about an OpenBSD much more effectively than you can about, say, a NixOS system. There is simply less that could surprise you.

It also means that OpenBSD is a solid rock upon which to build your church.

Most cloud engineers are by now familiar with the admonition to keep cattle and not pets. But one of the most important rationales for this rule has been the time it takes to recover a ‘pet’ from disaster.

The thing is, if your OS is not going to change much in the next ten years, it means that your shell scripts from ten years ago can do a great job of setting up a replacement for your deceased ‘cow’.

Meanwhile, cattle can come with overhead. Kubernetes, while beautiful and important, is a great example of just this: It allows you to treat nodes (instances) and pods (containers) as disposable, interchangeable units. But it does so at an enormous increase in complexity, and therefore, cognitive footprint.

If you’re a smaller shop (or a solopreneur,) Kubernetes might be A Lot to manage. This might in turn incline you towards one of the managed Kubernetes services that every Cloud provider wants to offer you, essentially getting you to shell out to have the complexity of Kubernetes (of keeping cattle) hidden from you. But this in turn locks you into a specific user interface provided by a specific vendor. Your skills aren’t transferrable anymore, and neither is your infra.

But! The mind-boggling diachronic stability of OpenBSD means that you can write shell scripts that will work as well in ten years as they work today, which in turn means that you can easily recover a failed OpenBSD instance with simple command-line maintenance tools. This in turn reduces your need for complex declarative-config machinery, like Kubernetes Deployments.

The churn resistance of OpenBSD means that smaller operators can put off ‘going K8S’, because OpenBSD makes it cheaper to keep cattle. And keeps those cattle more secure! Somehow unsurprising, given where it’s from :)

Now, the downside of OpenBSD is that hosting options are more limited, especially if you’re (say) an activist worried about the appalling 2023 legal environment. Fortunately, there’s an amazing option: OpenBSD Amsterdam, which sells (rents) OpenBSD virtual machines residing in, yes, you guessed it, Amsterdam, well outside the reach of scary American surveillance and British anti-cryptography legislation.

As luck would have it, OpenBSD Amsterdam is run by some amazing folks. Having used them for several years now, I can say that I have literally never had better service with any other provider.

So, that’s the basis for this website: Good old OpenBSD, hosted by OpenBSD Amsterdam, in Amsterdam.

Once you’re up and running on OpenBSD, the first step to setting up this website, after logging into the bare OpenBSD instance and putting my ssh key in ~/.ssh/authorized_keys, was to type:

pkg_add nginx git

This adds the NGINX webserver and git to your instance.

The next part of this series of articles will cover configuring NGINX and managing config with git on OpenBSD. This will get us to ‘hello world!’. Tune in next time!

Photo by Luc Viatour CC-A-SA-3

Calligraphic embellishment of the name 'Liz'