Before last week

I was away for two weeks, most of which was up in Scotland visiting family and friends, and staying at AirBnBs where they had animals - I can recommend this as a selection process for places to stay (others had a shetland pony, and sheep).

Last week

LIFE

Last week I was mostly focused on working with Alison get a final round of results for the biodiversity evaluation paper. This involved taking a set of the coarsened biodiversity maps for different species, and evaluating them for different versions of the evaluation metric (changing Z, for those familiar with it) and then also adding support for migratory species to the evaluation. Thankfully I had both Alison on hand and some work from Tom Ball that did similar things, and so I was able to compare the work I was doing to expand Alison’s scripts with those from Tom’s - a sort of virtual pair programming. By the Thursday we had Daintree generating the final half million TIFs Alison wanted.

From a broader perspective this work also draws to a conclusion the AoH computation line (at least for now), and so we have things ready to go into a pipeline similar to that Patrick has created for the TMFEMI: we in theory should be able to take everything in the persistence-calculator repository and go from IUCN data/Jung maps to these final summarised TIFs that Alison will make from all the ones I generated. That I suspect will become my task in November, particularly as it will align with potential work with Simon Tarr from the IUCN to help with their AoH pipeline.

The other fun thing about this was I did the entire week of work using the containerised development environment, using the fsark based arkpython3 and arklittlejohn for all my work rather than setting up a local virtualenv etc. It’s a small part of eating ones own dog food, but it was nice to note that this was actually just easier for me to do than worrying about GDAL working properly.

Tropical Moist Forest Evaluation Methodology Implementation

After two weeks PTO and a week on biodiversity work, I did slowly start to get back into TMFEMI work, which is still yet to conclude. I fixed an issue with Yirgacheffe, whereby it wasn’t coping with tiled datasets if you read off the top or bottom of the area covered by tiles.

EROFS

I had the briefest of play with EROFS, whereby I made a simple volume with EROFS and mounted it (all our machines now have erofs-utils installed). I was interested in this after Anil mentioned it as I like the idea of having a disk image file format that contains the outputs of one compute stage that can be used as the input to another that can collect many individual files (e.g, a batch of related rasters, or a Postgres database) but be passed around as an atomic blob.

EROFS though isn’t quite what I want yet:

• It’s read only, so we’d need to have some wrapper for it to collate outputs into a single blob
• I can’t just map it to runc

But, in theory I could make fsark do both these things.

Part III student

I had a chat with a Part III student who’s looking to do some work with iNaturalist data and linking it with some of the AoH work and using machine learning models. I gave them a pointer to some of the AoH papers.

This week

• Patrick and Robin have both given me things to look at regarding processing data in Yirgacheffe, so I want to get on that first
• Look into why FSARK isn’t mounting /home on one of our compute servers