Analysis paralysis, yak shaving, and all that
Published on July 11, 2020
I’m working on deploying https://mentor.alumniei.pt, and I decided to host it on a national VPS. It took me around a month from deciding to look for portuguese providers to actually pressing the “buy” button.
Now that I have access to the VPS, I need to configure it to host a rails application. I knew that I didn’t want to configure a systemd service manually. I knew I wanted something docker based. I had heard of dokku, so I decided to give it a go. It was easy enough to get it working, with the postgresql and letsencrypt plugins.
But during the installation process, I ran into a small issue: running
domains:set-global would yield “permission denied” accessing an arbitrary
file. The file was root owned, instead of dokku owned. a
chown solved the
problem and I moved on. But now I have this thing in my brain constantly
nagging me, telling me I should make a fresh installation in another server to
see if I can reproduce the problem, find the bug and fix it. I didn’t do this
yet, since I’m trying to focus on actually delivering the project, but I wasn’t
completely successful at preventing myself from being distracted by this
A couple of days ago, I noticed that their documentation, dokku.viewdocs.io/dokku, is not served via HTTPS. I never heard of viewdocs before, so I decided to investigate. I found a recent issue mentioning the lack of SSL. It seems like they’re hosting viewdocs on heroku, and since the project uses subdomains, it would require a wildcard SSL certificate. I searched a bit and found that this isn’t supported by heroku, reported it in the issue, and the issue was closed.
While I could have left it at that, it was still bothering me. So while I was
drafting this article, I came up with an alternative solution and suggested it
in a new issue. This was
prompted by the fact that I used
html-proofer to ensure that
every link still works and uses https, so now every time that I paste a link
onto an blog post article my brain automatically runs
html-proofer. I don’t
even have it set up on CI or anything. I just have the gem installed.
I’m not sure if viewdocs will ever add SSL support, but now every time that I check dokku’s documentation this will be in the background, nagging me.
Back to the application deployment process. To get something running, I had to install the postgresql plugin. Setting up SSL was easy enough with the letsencrypt plugin. But that’s not all that the application needs. It needs to send email (for registration confirmation, invitations, and password recovery) and it needs a storage solution for profile pictures.
To handle file storage I could easily use an S3 bucket, since activestorage supports it. But I am going through the trouble of finding a national VPS, doesn’t using S3 kind of defeat the purpose? I have more than enough disk space for a handful of profile pictures, and students are probably located in Portugal, so I don’t need any CDNs. I thought about setting up a volume for persistent storage, but it would be a manual process, and I wanted a solution that I could use in other self-hosted projects without much hassle. I remembered that min.io is a thing, but there’s no dokku plugin. So I started digging through dokku’s documentation on how to create a plugin.
I noticed that postgres and redis both use a concept named “services”,
represented by ~160 lines of almost identical code in their
functions file. I
started thinking about how I could extract this into a dokku concept, and went
through their issues to see if I could find something like that. I found
nothing. implementing min.io as a dokku plugin is probably going to be messy,
because it would also need its own domain and a letsencrypt certificate, and it
would just be a mess. I still haven’t decided how I’m going to tackle this, but
it’s probably going to be with a volume.
On the email side, my immediate reaction was that I couldn’t possibly self-host
email. Everyone knows that’s impossible. Porto codes even hosted a
presentation about it. SES is
easy enough to configure. But it requires AWS credentials, do I really want to
put personal AWS credentials on this server, and deal with their JSON policies?
This friction was enough to get me to entertain the idea of self-hosting. I
requested (and was granted) a PTR record on the VPS’s IP address. I started
postfix docker images, and if I could architect this as a dokku
plugin as well. Would I be able to run a single postfix instance that would
serve every dokku app? How does postfix handle multiple domains, how can I
restrict them to certain users? Should I start with another domain, to avoid
tainting the alumniei.pt domain?
Meanwhile, there’s another project I was thinking of moving here: Make or Break. It has an elixir backend and a react frontend. A few months ago I started writing a Dockerfile for the backend, but never got around to finish it. Recently, we decided to shut down the backend for a while, since we won’t be holding another event soon, and we want to save some money by shutting down the servers. This would mean moving half a dozen static websites somewhere else. I offered to host them in one of my servers.
This got me thinking if there’s a self hosted netlify solution as well that
could co-exist with dokku in this server. I could use dokku with the nginx
buildpack, but compiling and running
an nginx container per static website feels wasteful. I spent some time
searching, but no luck. I don’t even have a good idea of what a good
alternative would be. I have a bunch of static websites deployed on
hugopeixoto.net, but they’re all configured
manually. Every time I want to add a new one, I have to
letsencrypt certonly --webroot yadda yadda, ignore
the fact that this OS is so old that it still uses
letsencrypt instead of
certbot, copy and edit something to
symlink it to
service reload nginx. Maybe I’ll write a
create-website bash script that handles most of this for me and suppress the
stressful feeling that I’m moving more websites onto a half assed setup.
Dokku’s documentation states, somewhere, that it might not play well with other software that manages docker images. This could compromise my plans of having a netlify-like thing running, and even of having a postfix image running. Is dokku designed to work with non-web applications? How does dokku work? I didn’t search for long, but I couldn’t find an architecture diagram or something that explains what’s happening in the background. I expect that they have an nginx container that serves as a reverse proxy for installed apps. I know that plugins like postgres work via docker as well. Maybe I could help them with some documentation contributions.
One of dokku’s getting started pages mentions something in Vagrantfile. This note confused me for a bit. Does dokku use Vagrant under the hood? I didn’t change any Vagrantfiles, am I doing it wrong? Maybe by changing the way it is phrased, this confusion could be prevented. So now I have this task on the back of my head as well.
I also have this pokedex project tab open. My alpha sapphire save is in japanese, and I wanted to easily check what the english name of a move is, given its japanese name. First, I started crawling bulbapedia. Then I found this project, and started writing a command to list the learnset of a pokemon in a given game edition.
Then I noticed the readme (
The project is not dead, but it is languishing)
and the number of issues (115). There isn’t any Let’s go Pikachu and Eevee data
yet. It’s also missing Sword and Shield. So I tried to understand what’s their
favorite source of data, and it’s game dumps with manual fixes. So now I’m
looking at how to extract data from pokemon games, and also thinking of how to
handle changelogs, things that change from one generation to another. I joined
their IRC channel and offered my help for any low hanging fruit tasks.
I think I can detect these rabbit holes, but it’s not easy to get out of them. When it’s just one thing or two, it doesn’t bother me as much, but when every task unfolds into an attention grabbing black hole, I can’t get much done. It doesn’t help that every task that I try to tackle reminds me of another underlying problem that’s still not solved.
I’m not very good at note taking and organizing my time. Maybe this is what todo lists are for. This post was an attempt at getting all of these items out of my system, to see if I can focus on finishing something.
I’ll try to focus on getting mentor.alumniei.pt out of the door, even it if requires using SES during the first months. Moving the Make or Break websites is not a priority, we can pay for the servers for another month if necessary.