Last week we took our semi-regular trip to the Uffington White Horse, a 3000 year old horse carved into the chalk of a hillside somewhere between Swindon and Oxford. The horse is maintained by the National Trust, and each year they get a series of volunteer groups to come and help with the maintenance; we attend as part of the Long Now London cohort.

Perhaps it was the atypical heat getting to my brain, but whilst sat on the hill, trading in my keyboard for a trowel, I felt there were a good number of parallels to my time maintaining data-science pipelines, and possibly more broadly to software maintenance in general. This post is an attempt to collect those thoughts to see if they form any sort of cohesive comparison, or were just a sign I should have worn more sunscreen.

Constant maintenance required

Although we tend to think of geological features as being somewhat slow to change, the Uffington White Horse takes constant maintenance. Andy, the lead guide from the National Trust, was saying that it'd take only 5 to 10 years for the horse to vanish if it wasn't regularly maintained, which is a fate that has befallen many other such monuments. The type of maintenance varies year on year, as some tasks need to happen every year, such as the weeding, which is what I spent my time doing this year, versus laying on fresh chalk, which might happen every other year.

---

For data pipelines you might bet tempted to hope that once written, that's it: you can push the code to github once the experiment is finished, and when you next need it in a year or two for the follow on paper, it'll be good to go. Unfortunately that isn't the reality. No software package sits in isolation, we all build upon the work of others, and one of the unfortunate trends of development in the open source internet age is those things march on relentlessly when you're not looking.

Typically this will cause trouble if you don't specify the absolute version of every package you depend on: these will update and occasionally introduce changes that change an API or some behaviour you depend on. In general I'm against pinning every dependency to a tight version, as people forget to update them later and then you miss important bugfixes or security fixes, but if you're parking your code for a hiatus on some other project, then it's probably worth doing a pip freeze or whatever the equivalent is for your language of choice. But even doing that isn't a guarantee of reproducibility down the line: I've done this in the past and found that although I pinned all my direct dependencies, some of the packages they depend on were not properly pinned. Then in HPC world you have things like the version of slurm or the GPU drivers might have moved on since you last used them too.

Basically the pace of software tooling is constantly moving forward, and it's just an unfortunate fact that unmaintained software that works today may well break by the time you run it next year if you don't put in a little maintenance effort now and then.

Maintenance is multigenerational

Maintenance at this site has been going on longer than any individual contributor has been going on. Their motivations have been changing: at one point the local lord would get villagers to help maintain the grounds once a year and then throw a big party to compensate the villagers for their efforts, and now it's people less directly related to the area coming in from afar to help.

---

Even though the time scales are much shorter, in research software there is a similar problem as the code is typically worked on by PhD students and research associates/assistants who have fixed short term contracts, which means even over a five year project the code base could have changed from one generation to another. In both cases you hope there's some overlap to help hand over the techniques and local knowledge, but we on the software team need to assume that we won't be the future custodians of the code we work on today, and leave tests and documentation and other structures in place to ensure the next generation can succeed.

Maintenance is not the same as preservation

I think there is a reasonable but incorrect assumption that the point of maintenance is to leave a thing exactly as it was before. I overheard one of the archaeologists from Oxford University who happened to be on site saying that in an early era of maintenance there was a rule that when weeding you had to snip the weed at ground level, but never disturb the chalk on the monument, as that would be damaging to it. Anyone who has had to do any gardening or maintain a path or driveway on their property will know this is a bad way to control weeds, as they'll just immediately grow back: you have to remove the roots to have any hope to stop them returning and spreading.

And that is now what we do on the horse: I was given a trowel and instructed how to do a minimal dig to remove the weed with its roots, and then use the trowel to pat down and resurface the chalk so as not to leave a scar on the surface. In doing so I am modifying the horse ever so slightly, but in a way that ensures long term survival.

---

I'd say the same is true for maintaining data-science pipelines. Whilst in general I hope that if there's a breaking change to a dependency I just need to change one line here, there are times where perhaps a package is no longer maintained and we need to move to something else, or someone else has implemented part of the pipeline in a more performance orintated way or is doing more active maintenance. In which case, I'd advocate you should take advantage of that, and change the pipeline to follow the path that means we're sharing more code with others and following common practice in our domain.

To do this on a chalk horse is easier: I can look at where I've patted down the chalk again after removing the weed and know that I've done a good job. To do that on software requires a way to check that any change has left you with something that did what it did before, and so for that you ideally want a lot of unit tests or some other output check to be confident that you have maintained the pipeline.

Maintenance is active engagement with the work

Okay, so this year you're doing weeding. How hard can that be? You get a little tutorial on the trowel, and then look at where the horse should be chalk, and if you see a plant, remove it. Easy, right?

But it's not that simple. Yes, the weeds in the middle of a patch are quite easy to spot, but when you get to the edges, it gets a bit more murky: where has the grass and weeds encroached on the horse, and where has the chalk encroached on the grass, washed over by rain water or animals? There isn't a right or wrong answer here, you, the maintainer, have to make a decision. I could go and ask one of the National Trust people for every inch of horse edge, but do they know what it looked like last year? And is that what it looked like 100 years ago or 3000 years ago?

There is no perfect answer here: you, the maintainer, have to make that decision. Each weed you remove is an act of curation or design. You are choosing what is part of the horse and what isn't. Yes, if you get it wildly wrong one of the organisers might step in to stop you, but in general you are not just doing labour, you are part of the horse now, it is shaped by your decisions, as small as they might seem.

It perhaps isn't surprising that in general, despite the above decision making, people maintaining the horse tend towards a conservative approach, trying to make the least change possible. Over time that has led to the horse being thinner than it used to be: the grasses push on aggressively enough that the boundary still presses in over time. In the last few years archaeologists have studied the area around the horse and found that there is evidence to support it being widened a little bit all around.

---

Maintenance isn't as glamorous as creation unfortunately, and so attracts fewer people to it, but I think that's partly as it's not seen as being as mentally engaging, and that couldn't be further from the truth, and in some cases might need more engagement than creation, as creation wasn't constrained by consideration and respect for what went before. I think if perhaps we better communicated the level of thought and insight that even trivial maintenance requires then it might be more attractive to people.

In software the other lesson I take here is just that your job isn't to keep the code as it originally was: if best practice moves on, then follow it. The goal here is to keep the pipeline running and producing results, not to treat each line of code with reverence. The goal is maintenance of the method, not the current artefact that happens to embody that.

Too much of a good thing: part 1

When I started doing the regular maintenance on the horse, work would be done in two phases each year: a weeding phase, and then a chalking stage. The chalk from last year tends to discolour, so that is hammered into the ground to re-affirm the foundations of the horse, and then fresh chalk is poured on, leaving a nice clear horse. This hammering stage has become somewhat the focus of the volunteer effort as it's more fun than weeding. Indeed, the volunteer activity is usually just referred to as "chalking the horse".

But this year they had to undo some of this, as it's easier to add chalk in the middle of an area of horse, and less chalk was applied near the edges where it might roll onto the grass, and that bias has added up, giving the horse lines a convex rather than flat profile (in other words, it had a pot belly). So this year they had to remove some material to level the lines in the horse again, and now chalking is not considered an annual activity.

Andy was apologising to us in the pub after the first day, as this has been part of the draw of making it a fun day for volunteers, and if this stopped then how will they maintain the labour force for the duller bits? Us on the Long Now London crew were fairly unanimous in saying we were happy to do whatever jobs need doing, but you can see why it might be of concern to Andy how he balances the practical needs of the horse with making it engaging enough to get some volunteer labour.

---

For me the weeding is actually quite satisfying: you start on a patch of horse that has weeds on it, and as you go along you clear it and then you can look back at the area you worked on and see it's now a nice clean patch.

In software, fixing little bugs can be quite the chore, but I feel that by having a good suite of unit tests you can get to that same level of satisfaction: it comes from knowing that your effort did a change: either you fixed some tests that were red and now they're green, perhaps your coverage metrics went up, or you added entirely new tests and the test count is hire than before. I think as a mental game tests can help make little chores like fixing tiny bugs have at least the satisfaction of looking back down at the ground and seeing you cleared an area.

Too much of a good thing: part 2

Another issue in balancing the needs of the volunteer community and the needs of the horse is that Andy has something of a "success failure" on his hands: too many people want to maintain the horse now. Word of mouth has spread by people like me who went and had a good time: you get to sit on a hill, feel like you've contributed to something beyond yourself that will outlive you, and you get to hang out with the kind of person who thinks it's a good idea to sit on a hill and feel like you've contributed to something beyond yourself that will outlive you. So perhaps you have too many people chalking now that leads to you having to remove chalk the next year.

That is an idea that Chris, who is one of the Long Now London organisers, is grappling with as he has to work out how to focus this energy otherwhere. It's not like there are not other things in the world that need maintaining, it's just they're perhaps not 3000 years old yet. However, you only get to be something 3000 years old if someone maintained you at year 1, year 10, year 100, etc. How do we make that seem as appealing when the reward isn't within your own lifetime?

---

It's not much of a stretch to see a similar issue in open source software: there are big headline projects that people use all the time where being a maintainer might be useful for bragging rights or to improve your CV. We see maintainers of big projects being flooded by AI generated pull requests. But other smaller projects need maintenance too.

Over the last year I've made PRs on a number of small projects with few users that I happen to use, as I hope that it'll keep them going for me in the future, and perhaps help others find them useful, but that's still motivated by personal gain - I'm fixing a bug that's got in my way. How do we take the eagerness of new software engineers trying to gain experience and CV points and more widely distribute that to help maintain the broader commons?

Perhaps it's the solitary nature of coding that makes it harder to see how to bridge that gap. Real-world volunteering is in part a social thing: you get to do it with others. If I'm honest I do chalk the horse partially for selfish reasons too: it means I get to hang out with interesting people for a bit. I know from talking to others that they contribute to open source projects for that community spirit too. But I think both have a problem with bootstrapping community: there are many small open source projects that no longer have maintainers or just one maintainer for whom this isn't their main focus, yet would benefit from more regular updating; and in the real world for every Uffington White Horse there are many things in your local area that probably need someone to go maintain them but there is yet to be that community to draw people into it.

Tack

My thanks to the folk from the Long Now London group and the Nordic-RSE community who gave me feedback on this post, and thanks to Laura James for the photo of me at work at chalkface. And a hat tip to Richard Darst whose HPC Kitchen posts were partial inspiration for my musings here.