Weeknotes: 29th January 2024

Last week

LIFE visualisations

I made more progress with the visualisations for LIFE, adding more of the requested data layers and some captions about the graphs.

Screenshot 2024-01-29 at 13.21.51.png

For now I need to just add a link to the raw data behind some of the layers and then I think I can down tools on this until I get some specific feedback on what’s done.

The weakest part of the process here is that the data processing for the layers is somewhat disjoint from everything else, with some code going into a PR against Alison’s code for making the graphs for the papers to automate the manual QGIS operations that happened, some code being in the website repository, and some of it being built by hand using gdal2tiles.py.

Tropical Moist Forest Evaluation Methodology Implementation

I spent some time last week working with Abby, helping her get parts of the TMF pipeline up and running - this involved getting the documentation I wrote last week merged to the TMF repo, and adding some local server specific docs to our internal notion.

Given Abby’s the first person outside the compsci side to run this, we inevitably hit a series of hiccups along the way, but by the end of the week Abby had run a project area and got pairs for it, which is her current objective, repeating this for a bunch of Verra projects.

Things I had to fix along the way:

  • We use Amelia Holcomb’s biomass recovery project to back a bunch of our work, and I probably used it as a more general library than it was ever intended, and that caused some issues whereby to download elevation data you needed to specify the location of download cookies for GEDI data. Totally reasonable in the original context, but less good when you’re just trying to download random other datasets. There was one issue whereby if you install the project in a system context it’ll fail also due to expecting to be run from it’s own folder where it could create directories, which I did make a fix for. I say, this is because we’re taking code designed for one particular use case and changing it to be more general and just finding some edge cases there, so it’s on us to fix these as we find them.
  • My container abstraction layer, fsark, didn’t let you specify environmental variables from the outside, which was the workaround for the above, so I added that, which is a useful general improvement to fsark.
  • In addition, fsark didn’t support network access. Mostly this is because I’d not added support for it, but in general I see this as a feature, as longer term I’d like to force downloads through a proxy to make provenance easier to track. Still, to unblock Abby I tweaked fsark on how obuilder works to support downloading for now (hat tip to Patrick Ferris for pointing me at how obuilder does things).

Whilst this took time to get through, it was all useful progress in making the TMFEMI pipeline more robust in being used by others outside the original development team, so it was good to see happen.

This week

  • TMFEMI - Deal with any issues that Abby encounters.
  • LIFE - Add a link to download the raw data, and get that data into the data container image
  • Shark - I need to pull together some thoughts on how fsark, pyshark and Anil's latest ideas fit together as an end-to-end provenance solution.

Interesting links

  • I’m about half way through a podcast on Fixing the messy voluntary carbon market on how carbon markets might be made better, and am thus far enjoying it. Kinda wish it had existed when I started - certainly Catalyst podcast is now on my regular listening playlist.

Tags: weeknotes, life, tmfemi, fsark