Things I did last week

Tropical Moist Forest Evaluation Methodology Implementation

I did another pass through both the Counterfactual Methodology document and the code that Tom/Miranda had been working on, and identified all the inputs needed, and where we got them from, in particular noting both ones I didn’t have access to, as they’re either in Google Earth Engine (GEE) or local files not yet shared with me/Patrick, and ones in the code that don’t show up in the methodology.

Containerising things for Ark

As outlined last week, I spent some time writing a little wrapper for both littlejohn and python that runs them seamlessly (at least in theory) in a container instance that has all the correct packages required for the persistence work. This needs a little better packaging, but otherwise is at the stage where I want to try testing it with the ecology crew. If this works then I’ll also extend it to work with R. The benefits here are:

• We have regular problems with pip and GDAL libraries on our compute servers, due to the older fixed version of GDAL on the machine due to Ubuntu LTS, and then the pip versions having issues for that older release
• It means we can start to have a repeatable environment for runs that we can trace back to a specific container as per the Ark dream
• At the command line though it still looks like the same old tools the ecologists are used to, just they don’t need to use a virtualenv, and there’s a lot more faff when they want to add a new package for now :)

Machine admin

• I had to set up CUDA on our compute server with GPU for one user
• I’m having to nag people about watching their memory usage on 1TB RAM machines - I feel that there’s some future work for littlejohn where it watches dynamic memory usage and suspects children when the machine us under memory pressure (I’m getting Bromium flashbacks here). Whilst Yirgacheffe does better memory management under the hood for our python code, not everyone’s tasks are able to use it (In R or external GDAL tools).

To do this week

• Catch up with Patrick on TMF work to find out what I can pick up
• Tom gave me access to some GEE data needed for the TMFEMI work, so I need to make sure I can access that.
• Get the container wrapper tool installed on all our compute servers
• Deal with all the compute server admin requests that seem to be flooding in

Interesting links

• Three years in Antarctica - affordable and durable time-lapse monitoring
• Dagster - a data pipelining/cloud orchestration tool, closest I’ve seen yet to an Ark like thing

Interesting lynx

I was away the first half of this week at a wildlife photography course at Wildwood Rewilding centre, enabling this pun.