Analysis paralysis, yak shaving, and all that

Published on July 11, 2020

I’m working on deploying https://mentor.alumniei.pt, and I decided to host it on a national VPS. It took me around a month from deciding to look for portuguese providers to actually pressing the “buy” button.

Now that I have access to the VPS, I need to configure it to host a rails application. I knew that I didn’t want to configure a systemd service manually. I knew I wanted something docker based. I had heard of dokku, so I decided to give it a go. It was easy enough to get it working, with the postgresql and letsencrypt plugins.

But during the installation process, I ran into a small issue: running dokku domains:set-global would yield “permission denied” accessing an arbitrary file. The file was root owned, instead of dokku owned. a chown solved the problem and I moved on. But now I have this thing in my brain constantly nagging me, telling me I should make a fresh installation in another server to see if I can reproduce the problem, find the bug and fix it. I didn’t do this yet, since I’m trying to focus on actually delivering the project, but I wasn’t completely successful at preventing myself from being distracted by this

A couple of days ago, I noticed that their documentation, dokku.viewdocs.io/dokku, is not served via HTTPS. I never heard of viewdocs before, so I decided to investigate. I found a recent issue mentioning the lack of SSL. It seems like they’re hosting viewdocs on heroku, and since the project uses subdomains, it would require a wildcard SSL certificate. I searched a bit and found that this isn’t supported by heroku, reported it in the issue, and the issue was closed.

While I could have left it at that, it was still bothering me. So while I was drafting this article, I came up with an alternative solution and suggested it in a new issue. This was prompted by the fact that I used html-proofer to ensure that every link still works and uses https, so now every time that I paste a link onto an blog post article my brain automatically runs html-proofer. I don’t even have it set up on CI or anything. I just have the gem installed.

I’m not sure if viewdocs will ever add SSL support, but now every time that I check dokku’s documentation this will be in the background, nagging me.

Back to the application deployment process. To get something running, I had to install the postgresql plugin. Setting up SSL was easy enough with the letsencrypt plugin. But that’s not all that the application needs. It needs to send email (for registration confirmation, invitations, and password recovery) and it needs a storage solution for profile pictures.

To handle file storage I could easily use an S3 bucket, since activestorage supports it. But I am going through the trouble of finding a national VPS, doesn’t using S3 kind of defeat the purpose? I have more than enough disk space for a handful of profile pictures, and students are probably located in Portugal, so I don’t need any CDNs. I thought about setting up a volume for persistent storage, but it would be a manual process, and I wanted a solution that I could use in other self-hosted projects without much hassle. I remembered that min.io is a thing, but there’s no dokku plugin. So I started digging through dokku’s documentation on how to create a plugin.

I noticed that postgres and redis both use a concept named “services”, represented by ~160 lines of almost identical code in their functions file. I started thinking about how I could extract this into a dokku concept, and went through their issues to see if I could find something like that. I found nothing. implementing min.io as a dokku plugin is probably going to be messy, because it would also need its own domain and a letsencrypt certificate, and it would just be a mess. I still haven’t decided how I’m going to tackle this, but it’s probably going to be with a volume.

On the email side, my immediate reaction was that I couldn’t possibly self-host email. Everyone knows that’s impossible. Porto codes even hosted a presentation about it. SES is easy enough to configure. But it requires AWS credentials, do I really want to put personal AWS credentials on this server, and deal with their JSON policies? This friction was enough to get me to entertain the idea of self-hosting. I requested (and was granted) a PTR record on the VPS’s IP address. I started looking for postfix docker images, and if I could architect this as a dokku plugin as well. Would I be able to run a single postfix instance that would serve every dokku app? How does postfix handle multiple domains, how can I restrict them to certain users? Should I start with another domain, to avoid tainting the alumniei.pt domain?

Meanwhile, there’s another project I was thinking of moving here: Make or Break. It has an elixir backend and a react frontend. A few months ago I started writing a Dockerfile for the backend, but never got around to finish it. Recently, we decided to shut down the backend for a while, since we won’t be holding another event soon, and we want to save some money by shutting down the servers. This would mean moving half a dozen static websites somewhere else. I offered to host them in one of my servers.

This got me thinking if there’s a self hosted netlify solution as well that could co-exist with dokku in this server. I could use dokku with the nginx buildpack, but compiling and running an nginx container per static website feels wasteful. I spent some time searching, but no luck. I don’t even have a good idea of what a good alternative would be. I have a bunch of static websites deployed on hugopeixoto.net, but they’re all configured manually. Every time I want to add a new one, I have to mkdir /srv/web/domainname, run letsencrypt certonly --webroot yadda yadda, ignore the fact that this OS is so old that it still uses letsencrypt instead of certbot, copy and edit something to /etc/nginx/sites-available/domainname, symlink it to sites-enabled, and service reload nginx. Maybe I’ll write a create-website bash script that handles most of this for me and suppress the stressful feeling that I’m moving more websites onto a half assed setup.

Dokku’s documentation states, somewhere, that it might not play well with other software that manages docker images. This could compromise my plans of having a netlify-like thing running, and even of having a postfix image running. Is dokku designed to work with non-web applications? How does dokku work? I didn’t search for long, but I couldn’t find an architecture diagram or something that explains what’s happening in the background. I expect that they have an nginx container that serves as a reverse proxy for installed apps. I know that plugins like postgres work via docker as well. Maybe I could help them with some documentation contributions.

One of dokku’s getting started pages mentions something in Vagrantfile. This note confused me for a bit. Does dokku use Vagrant under the hood? I didn’t change any Vagrantfiles, am I doing it wrong? Maybe by changing the way it is phrased, this confusion could be prevented. So now I have this task on the back of my head as well.

I also have this pokedex project tab open. My alpha sapphire save is in japanese, and I wanted to easily check what the english name of a move is, given its japanese name. First, I started crawling bulbapedia. Then I found this project, and started writing a command to list the learnset of a pokemon in a given game edition.

Then I noticed the readme (The project is not dead, but it is languishing) and the number of issues (115). There isn’t any Let’s go Pikachu and Eevee data yet. It’s also missing Sword and Shield. So I tried to understand what’s their favorite source of data, and it’s game dumps with manual fixes. So now I’m looking at how to extract data from pokemon games, and also thinking of how to handle changelogs, things that change from one generation to another. I joined their IRC channel and offered my help for any low hanging fruit tasks.

I think I can detect these rabbit holes, but it’s not easy to get out of them. When it’s just one thing or two, it doesn’t bother me as much, but when every task unfolds into an attention grabbing black hole, I can’t get much done. It doesn’t help that every task that I try to tackle reminds me of another underlying problem that’s still not solved.

I’m not very good at note taking and organizing my time. Maybe this is what todo lists are for. This post was an attempt at getting all of these items out of my system, to see if I can focus on finishing something.

I’ll try to focus on getting mentor.alumniei.pt out of the door, even it if requires using SES during the first months. Moving the Make or Break websites is not a priority, we can pay for the servers for another month if necessary.