Tech

Dr Jekyll and Mr... Hugo?

In December, I set out to move my site from the previous static site generator, Jekyll, to a brand new one, Hugo. Perhaps inevitably, this job turned out to be a lot more work than I expected. Equally inevitably, I wildly overestimated my enthusiasm for hacking away at this over Christmas. As such, here we are at the start of February and it’s (just about) finished.

So what’s changed?

Static Site Generator

The site was previously put together using Jekyll. This is a ruby-based static site generator that takes a heap of markdown files, mashes them through a bunch of templates and generates some HTML.

The new static site generator, Hugo, does more or less the same, but is written in Go. So why change? Well… mostly for the fun of doing so, if I’m honest. There’s a bunch of advantages (I can write Go, I can’t write Ruby; it’s slightly faster; it’s got a large current dev community behind it) but none are so big as to really merit this change.

Infrastructure

I’ve also used this change to move to running this site with Infrastructure-as-code at the heart of the setup. It’s hosted in AWS Amplify, which is astonishingly easy to setup, and managed by Pulumi. There are a few benefits here:

• It was fun to do
• The setup can be read in plain text - no clicking around a hundred different AWS portals
• Changes are easy to make, audit and revert
• The code for doing so now lives alongside the site in the public repo

The only necessary code or config that isn’t in the repo now is the DNS settings. These are managed in Cloudflare using Terraform/Pulumi. They’re only separate because they live alongside a bunch of other, unrelated DNS stuff.

Chopping Mall

Over at /chopping-mall is something I’m very pleased to have restored: my first blog. I wrote on Chopping Mall intermittently for several years ~2010. It’s always bothered me that it was held in Blogspot / Blogger with no easy way to export it. But no more! I found some XML export and Hugo import tools and now have all the old posts in Markdown, in the repo. There are some broken links and images to sort through, but the vast majority is there.

Walks as its own section

I’ve split out /walks into its own dedicated site section. Still with maps. Still with files you can download. I’m edging slowly away from Geojson file (lovely open format, yay! Not exported by OS Maps app, boo!) to GPX files.

More stuff to come…

My hope is that my renewed enthusiasm for this blog will get me using it to build up interesting things bit-by-bit. I’m intending to do booklists, filmlists and so on, and generally use it a bit more. Let’s see…

How to build the new music library

Having decided that it’s the right time to rebuild my music library (see the previous post), I’m now starting to scratch my head over how to do it.

There are a few areas I clearly need to get my head around and consider the option available to me. These include:

• Software: how am I going to organise my library? What will I use to manage tagging files? How will I add files?
• Data: how visible will my library data be? How easily can I query it, explore it, check it? This is going to get important when it comes to considering quality control, missing releases, spotting duplicates and so on.
• Process: how am I actually going to work through the process of building this library? Where do I start? What do I prioritise?
• Goals: What do I actually want to be able to do, other than listen to the music (obviously!). Am I looking to build themed (sub)collections? To explore genres over time? To listen to specific record labels?

Some of these are easier to answer than others.

Goals

I’m trying not to overcomplicate this, but I think there are quite a lot of exciting opportunities here. I want to be able to:

• Listen to a specific album (obviously)
• Listen to a specific artist
• Listen to a specific genre
• Listen to a specific label

I’d like to be able to build and amend ad-hoc collections with ease (“music for driving”, “cooking music”, “ambient-ish-stuff-to-listen-to-while-reading”).

I’d like to be able to visualise all of the above. I love a graph.

A lesser goal - perhaps more of a nice-to-have - would be to have linking between artists, records, labels, genres. Something where you can get to similar things and explore your collection. I have no idea how feasible this is, and it’s by no means essential.

Data

I want full access to my data. I’m taking it for granted that whatever software I end up using is going to add some sort of data layer on top of file metadata tags - otherwise any query would have to read the whole library! This data must be easily accessible. I don’t mind if its a simple database written to disk or a JSON API that I can query, I just need to be able to get hold of the data independently of the library software.

The simpler, the better.

Software

I think I’m confident in my choices here. I’m sticking with Beets.

Beets has served me well for many, many years. It’s a solid bit of open source software built by music nerds for music nerds. It slurps in new files, tags them according to rules and organises them on disk. It’s backed by a Sqlite database file, which makes querying the data yourself really easy (apart from Sqlite being a pig about concurrent access…).

Process

I think I need to get the basic technical elements set up first. Something along the lines of:

• Decide where the files live
• Create a new beets library
• Build a sufficiently comprehensive beets config (consider tagging, file paths, album art, etc)
• Consider how I access the library (web ui, db export, MPD, etc)
• Import an initial batch of albums to see the process in action

After that, I think the fun really starts. It doesn’t actually matter how I approach it, I just need to keep adding to the library. I plan to do weeks where I explore a specific genre, the back catalog of a specific artist, revisit records I’ve not listened to in a while, explore something brand new. It’s all welcome.

Key decisions that need making:

• What are the tagging rules?
• What’s the filepath structure?
• What do I do with album art?

Other things to explore:

• Are there nice beets plugins for browsing a library?
• Has anyone worked on mapping MusicBrainz IDs to Spotify URNs?
• Should I commit the configs to a GitHub repo somewhere to go alongside these posts?

(Re)building a music library

It’s coming up to the end of the year, a time for making plans, resolutions, projects and so on and so forth. How many of them will survive contact with 2025? Who knows. But here’s the tentative start of a new project: I’m going to rebuild my music library. And I’m going to document the process on here.

But why though?

The project? Well… I have librarian blood. It goes deep into who I am. Stuff’s got to be organised! And it really, really isn’t. I’ll get into the detail of the problem further down this post but it’s got to the point where it makes me twitch. Something must be done!

There are also loads of sensible, pragmatic reasons. Not least:

• Spotify (and equivalents) are fragile, capricious and temporary: Your favourite album might disappear next week. Your playlist might be missing key tracks. You don’t own this. It can be taken away.
• Size and space: I’m quite good at backing things up and keeping spare copies of digital information. Storage is cheap, but it’s not that cheap. It makes sense to keep only the stuff you want in your library if you’re paying to keep 3+ copies of it!
• Rediscovering old gems. A large part of this project was spurred on by the simple game of listening to every one of my vinyl records. We moved house, I got my record player set up, I got my records out of storage, and I began to work my way through relistening to each one. I found so much stuff that I’d forgotten about. Half of the point of this project isn’t really about the organisation, it’s about the increased engagement with the music.

How did we get here?

I was lucky enough to be really getting into music right at the tipping point from physical to digital media. All my first albums were on CD but we were busy ripping them to MP3 to write to a new disk, title scrawled on in marker. We were grabbing the early Arctic Monkeys demos from MySpace. I was posting on music forums, with mailing lists where we forwarded whole albums to each other (RIP AudioJunkies - much missed). I was testing how well iTunes’s windows app worked on a sprawling library (not well, it turns out, prompting a swap to FooBar2000).

Music was something to be explored and studied, and in doing so acquired. Before we had (semi-)reliable digital libraries, you had no idea if you’d ever hear that track again if it didn’t get added to your library. Right-click. Save-as.

So the library grew and grew.

How bad is the problem?

It’s really quite bad. My music library (heap?) appears to be 1TB in size, with roughly 76,000 files in it. This breaks down as 46k MP3s and 26k FLACs. That’s a lot of stuff. And I suspect that the vast majority of it isn’t stuff that I actually really want.

What can we do about it?

Start over. Sort of.

The plan from here is to build a new library. I’m going to curate it carefully from the ground-up, adding artists I like, labels I like, filling gaps where I find then. It’s going to be immaculately tagged and indexed, making exploring it a pleasure, not a chore.

It’s going to lossless (predominantly? totally?). It’s going to be organised. It’s going to be beautiful.

Quite how I do this will need some thinking about and another blog post.

How this all works

I’ve not done a how-it-all-works post on here before. Truth be told, quite often I’d probably have struggled to tell you how it worked. Some nice ideas, often half-implemented, a distraction or two, and somehow there it was. Until it wasn’t… I’d regularly forget to sort out SSL certs, because I’d never got round to automating it. And writing a new post was surprisingly hard. Where did I have to put the files? What makes it build? And suddenly I’ve lost the will to work on it and gone off to do something else.

But no more! No. For I have re-organised, redesigned and reimplemented this site from top to bottom. So here’s a whistlestop tour through the web-stack.

September 2023 update: Yikes! This post is already outdated. I’ve started moving to automatically triggered AWS builds which seems like an exciting step!

Site code

The content of the site is essentially a bunch of text files. They’re plain text, with formatting done in Markdown. This makes them very simple to write. To make them look a little better than plain text though, and sort out all the pesky HTML we lean on Jekyll. There are heaps of static site generators out there. I first discovered Jekyll years ago so have largely stuck with it because its what I know.

When moving the site around I did toy with jumping to Hugo - it looks like a more modern sibling to Jekyll with a lot less code in ruby - but in the end I couldn’t really justify the work to learn a new framework. Most of the tangible benefits seemed to be in the build times. I do believe it is faster. But my site builds in Jekyll in … checks console … 0.53 seconds. Build time is just not a problem I need to fix right now.

Having run Jekyll, all of my text files are transformed into lovely HTML. I check my new post looks like it should and then its time to push the files up to the webserver. And here’s where the next big changes come in…

Containerise everything!

My webserver is a cheapish VPS. It’s not particularly well resourced but its sufficient for this site. Previously all the config for serving up my site was sat around on the VPS in a bunch of different files. This caused a few problems:

• Testing anything was basically impossible, there was no dev server at all.
• Migrating between VPS setups was an absolute pain. You could all but guarantee nothing would work.

So once we had the problems, how to solve them? Containers. I’ve now migrated the entire setup (this and a couple of other sites) into a handful of Docker containers, all held together by Docker Compose. It’s neat. It’s so neat I wish I’d done it years ago.

SSL Termination

One of the first good steps I took was to split out SSL termination. This means I’ve got one container, the one that’s exposed to the outside world and its whole job is to serve up SSL certificates for all three sites, and then just pass on requests to the relevant site’s container. Currently, this is running NGINX but I’ve got half an eye on replacing it with HAProxy.

The real joy of splitting out the SSL functionality is that I can run the actual sites themselves entirely without certificates. This makes testing changes much easier. I don’t need separate dev or localhost configs to run dev versions of these sites now - I simply start the container up anywhere and it just works exactly the same as the real site.

The rest

The other containers are largely very simple. The one that serves up this site is another very simple NGINX image with the HTML files from earlier injected as a volume. Sorted.

Version controlled config

And it’s all under git. Finally, I can test changes on a dev setup, commit them to the repo and then just run a pill, build and relaunch command to deploy my site changes. And be confident that they work.

A cloudy future?

Earlier on I mentioned the difficulty of migrating this setup. I’m now relatively confident that this is a solved problem - the whole site is now just a bunch of config and static files, all version controlled and easy to check out. This should make moving to any other VPS an absolute breeze. The exciting alternative though is to explore some of the ‘serverless’ options out there - AWS, Azure, Google Cloud, etc. I’m not very experience with any of them but given that the basic modus operandi is spinning up Docker containers with the server-layer abstracted away, I’d say the site was primed ready to make the switch. Exciting times.

And lastly, here’s a bonus picture of “the internet illustrated in the style of The Matrix”, for no better reason than I really like it. You’re welcome.

All images generated with Midjourney

Why the internet is broken

The internet is garbage

The internet is garbage. Nearly all of it is awful. There are still pockets of good all over the place but the overwhelming trend towards big tech platforms, trapping you in walled gardens and exerting monopolistic power is incredibly depressing. What started out as a utopian dream now seems much more like a dystopian vision. Black Mirror episodes that once seemed outlandish are now basically history lessons.

It’s money. The relentless desire to monetise everything has pushed us down this road. For a good portion of Web 1.0 the internet was weirdly blasé about money. Services were generally free. People made stuff for the sake of it, out of interest or curiosity.

SEO has broken everything

Nowadays it’s all plastered with ads. They’re everywhere. And the places that aren’t actively ramming autoplaying video ads down your throat are busy collecting data about what you’re interested in so they can sell it to someone else to force feed you ads.

The inherent competition in selling ads (there’s only so much time anyone spends looking at the internet) means the push to be top of the search page heap and trample all your competitors is all consuming. SEO is a nightmare. Search engines are broken. Google is objectively much less useful for finding information on than it was a decade ago. Is this what progress looks like?

Communities are harder to find

One of the real paradoxes of how the internet has changed over the last 15 years is how much harder social media has made it to find real communities. The relentless drive to make every platform bigger and to suck in more and more users has torn apart the many weird little communities that spent years just doing their own thing. I spent years on forums: music, film, food, books - niche little worlds where people chattered away. Reddit, twitter and Facebook are a poor alternative. All 3 are clearly ailing and beginning to fall apart but it’s not at all clear what replaces them.

AI is the next wave of awful

As if all this wasn’t enough, here comes GPT. Why wade through tonnes of lazily churned out SEO optimised copy when you could instead drown in gigatonnes of AI generated SEO optimised copy? This is the future and it doesn’t look like the search engines are ready for it. If you thought that finding information was hard now, just wait till 90% of it is AI hallucinations lies.

Human voices

Unsurprisingly, I don’t have the answers to any of these problems. I’ve no idea how this is going to pan out. However it goes, though, I feel like the need for human voices, for recommendations, for curation, for selection, is only going to become more important. The AI generated playlists on Spotify are boring. The Netflix recommendation algorithm is terrible. And who’s going to trust an AI to review a restaurant?

As we drown in a sea of algorithmically generated junk, it feels inevitable that we’re going to long for human voices. But will it be too late?

All images generated with Midjourney (because of course they are!)