Coping with the loss of a loved server

On Sunday I got one of those chat messages one doesn't want to read: the SSD in the big machine I've been using for the habitat work was severely degraded, and so work was stopped on that machine, and I had to find a new home for my habitat map work.

For the last year or so most of the work I've done on biodiversity pipelines I've done on the Mac Studio that sits on my desk: with 32 CPU cores, 80 GPU cores, and half a terabyte of RAM this machine is no slouch. At the same time calculating global biodivesity metrics for tens of thousands of species at 100m per pixel is a tough job for any computer. At 100m per pixel a global raster is 150 GB in size, and so 34k * 4 layers at 150GB * multiple scenarios is a lot of data to chew through. But thankfully the work in these metrics is very GPU friendly, and because Yirgacheffe supports Apple Silicon GPUs thanks to MLX, the Mac Studio was the fastest machine I have access to for this particular workload. For the habitat map work though, I had moved to using our largest AMD EPYC server, which sports 768 CPU cores, no GPU cores, and 3TB of RAM. There were two reasons for this. The main reason when I started was access to Tessera, which as a 10m-per-pixel model with 128 layers in it, takes a serious amount of storage, so I needed to be on the University network where we had enough of Tessera. The other reason is that the naive machine learning code I was writing was much better to CPU parallelism than my GPU heavy but CPU light Mac Studio.

However, a machine with a (currently) working SSD is better than one with no SSD, so the start of my week was trying to move the habitat work to my Mac Studio.

Firstly, this is good disaster recovery practice. It's been a while since I had a disk fail on me, but it's certainly not the first time, so I like to think I was reasonably prepared for this. Everything is pushed to github, and I build my pipelines in general to not rely of random data downloads, but rather the pipeline scripts have in them fetching data from primary sources. But perhaps predictably I'd missed some things. For instance, I recently added a dependancy on biome data from the RESOLVE ecoregions dataset, and I wasn't sure that idea would work, so I just took a copy I already had, and then failed to add the download to the snakemake rules for the pipeline once I had decided that idea worked. The more interesting gotchya was that around some scripts for tidying up eBird data. I've mentioned before I think that this is the one project where I'm using Claude Code as the primary code interface, and how I'm trying to understand how one can use a tool like this and still have any relationship to the finished artefact. I general this has been working okay, but I found a couple of places where for data we have to manually download (i.e., the eBird database), which required a little pre-processing for our pipeline, those pre-processing scripts had not been added to the regular code base, Claude Code had just used temporary scripts to do a one off job, and those were lost. This error is totally on me, Claude Code is just a tool, and in general I do not assume it has done the right thing, but I missed these ones and had to pay the penance of recreating them myself.

I say recreating them myself, because moving to the Mac Studio has meant I've not been using Claude Code, as I don't trust it in auto mode outside of a Docker container or some other sandbox, but if I'm to hope to get any performance out of the pipeline on my Mac Studio I'll need access to the GPUs, which I can't currently do that way. Thus this week I've been playing with the pipeline directly again, and that has been a nice confirmation that my strategy of getting Claude Code to build many small scripts that do a single transformation in the pipeline, and getting it to follow my own coding style preferences has made it easy to transition to manual control.

On the topic of performance, I had to make two changes. The first was a physical change to my working environment, which involves an ethernet cable now going out of my office window, down one floor and across one room, and then into another window where the broadband router is:

Basically the walls of our house are sufficiently thick that I don't get the best wifi reception, and my attempt to use homeplug style powerline ethernet has not been any better. But for most of what I do the performance has been okay, so I've not bothered to fix it. But suddenly I needed more performance to let me stream Tessera embeddings from S3, and so I finally did what I've been joking about with my partner for a while, and throw a cable out the window of my office and into our living room. Suddenly I went from around 100 megabit to 1.6 gigabit, and what initially looked like it'd take 24 hours to just stream the UK data for Tessera suddenly took a tractable amount of time. Obviously this isn't a long term solution, but it is worryingly close to one of those temporary solutions that works well enough that it'll still be there this time next year if I'm not careful.

Tangentially, it was a nice reminder also that, being somewhat old, my head was still stuck in the "gigabit ethernet is fast ethernet" - I realised this because all our ethernet switches in the house are 1 gigabit max and I ran out of ports on the broadband router. Let it be known that I prioritised habitat map production over my Xbox connection :) I fear a set of new home ethernet switches will be on the xmas shopping list.

Back to the habitat map work, the other change was to try get the pipeline to use some GPU on this Mac Studio. Thankfully someone has made mlx-learn an MLX accelerated version of scikit-learn. With that I have the pipeline running at a usable speed on my Mac Studio, though not as fast as the AMD EPYC server, because this pipeline just isn't as GPU formatted as my other pipelines.

So whilst not a simple switchover, overall I think I survived the sudden loss of my regular compute environment pretty well. Now, a week later, the AMD EPYC server is back in action, so I will be switching back I think, as it lets me iterate much quicker, but a week reset was a good reality check for both how robust my code management at arms length has been, and where some of the performance bottlenecks are.

ETOOMANYTWITCHERS

On the science side of the habitat maps, the main thing I hit up against was just how unevenly distributed datasets around species data are. To give you a sense of the problem here's a heatmap of eBird observations from 2025:

This density of data actually gives us two distinct but linked challenges: firstly we're data starved in many parts of the world (particularly those ones we need to protect the most), and we have (for my purposes) too much data in data rich parts of the world.

The challenge is highlighted for me in my habitat maps when I try to use this data to validate my output. In the UK my map scores around 75% validation currently without eBird data, but that drops to around 60% with eBird data. Digging into this it's because in the UK we don't just record the regular species, but we record all of the outliers too - birds that have wondered out of their usual area. Indeed, bird watchers will be particularly keen to spot atypical species, so are perversely motivated to find occurrences that go against the general trend. In general, this is good recording for science, particularly as climate impacts disrupt habitats, but for a map like mine it becomes problematic. Obviously there's things I can do to thin down the data, but I need to be careful because if I apply those same rules to Madagascar or such then we start to remove data from what is already a very limited data set.

I've been trying to find guidance from what other people have done in this space, as I feel this can't be a novel problem. The best I've found thus far is this paper by Johnston et al 2020, though like much in the occurrence space I feel this is aimed at dealing with areas where there is too little data rather than too much (and the tension when you have both). Thankfully it turns out that the first author of that paper, Alison Johnston, is visiting Cambridge next week to give the CRI Seminar and so I've booked a slot to talk to her about the problem and see if she can point me in the right direction. For now I'm just going to take the poor score in the UK on the chin and move on focussing on the areas we care about more where having the eBird data helps more than it hinders.

Alpaca Festival submission

After I havered about it too long, I finally caught up with Patrick Ferris and we decided to do a workshop submission to Alpaca Festival, based around trying to teach functional programming concepts as they apply to algorithmic art. This is something that I've been interested in ever since I started messing around with graphics programming in OCaml with my Claudius graphics library. You can see me talk about Claudius and about how a lot of the demo style effects I make in it are inherently functional in style in my FunOCaml talk from a couple of years ago:

Back in August I took a stab at porting Claudius from SDL to Javascript, using JS of OCaml, which lets you run OCaml code in a browser by compiling the OCaml byte code format to Javascript. This worked, despite a few rough edges. This then inspired Patrick who has been playing with getting OCaml playgrounds to run in a browser to make one that works with Claudius as it's main feature:

This in turn motivated me to improve of the JS port, adding some simpler APIs to make the playground code no longer expose the HTML canvas lookup, and bam, we have the premise for a nice tool that is both useful for trying to teach OCaml and functional programming methodologies to artists, and also perhaps encouraging new people to try OCaml thanks to the fun and interactive nature of doing graphics programming.

There's a bunch of polish still required before we put this in front of a wider audience, but Alpaca Festival is in May next year, so we have time to make it more slick. We have a rudimentary save feature that needs tweaking, and I'd like to have a way to save the compiled JS so you can embed it in your own web pages to show off your creations. I'd also like to work in all the examples into the playground to give some different starting points.

Still, the submission is in, so now it's crossing fingers until we get old if we're successful or not.

30 day map challenge

I'll probably regret this, but I have started 30 Day Map Challenge - making a map a day based on the prompts provided. I tried this a couple of years ago an quickly aborted it due to challenges with open data that wasnt' really open making it hard to share progress. This year I have an side-project that this somewhat aligns with, so I'm going to use this as a way to chip away a little each day towards that. This means probably the maps for a given day won't be amazing, but at some point I can hopefully pull it all together into the proper side-quest thing and everyone will be "wow, now I see what you were doing". Or perhaps not.

The Day 1 prompt was "points", and for that I simply plotted occurrence data from GBIF that I was using for the habitat map work on a map.

The map is small, because the side-project demands it to be so, but you can hopefully still make out the area around North Wales, the Wirral, and Liverpool up to Blackpool that it is showing. This is knocked up in OCaml using Claudius, as is my jam, and it was a fun excuse to try out Patrick's OCaml bindings for PROJ, which is the standard map projection conversion library, as I'm drawing things in British National Grid (EPSG:27700) rather than using some mercator style projection.

The one thing I did think was interesting about the map is how Liverpool City Centre disappears - not much in the way of species observations in the city centre, despite urban as a category making up a significant proportion of species observations in the UK.

This week

• With the AMD EPYC server back, a return to habitat map progress. Whilst having to do some house keeping was good, it's put me back a week which is less good.
• Fix my tangled knot: there was a major update to Tangled last week, including switching their tooling from Go to Rust, which means I can no longer build the knot software on my VPS, despite it having 4GB of RAM 🤦 Thankfully Akshay Oppiliappan has said there will be binary packages soon.
• Ping people to let them know I'll be in Cambridge next week to set up meeting slots!