I put out what I hope might be the last 1.x release of Yirgacheffe this week, which is release 1.13. I realised in the middle of last year that I wanted to make significant changes to the Yirgacheffe APIs to make them less "computer sciency" and more "data sciency", and at the same time fill in some functionality gaps, and I bundled these up into a version 2.0 in my mind, because it would necessitate some breaking API changes. However, over the last eight or so months I've managed to do most of this without breaking any existing code by adding the new APIs alongside the old ones, and I've got much closer to a 2.0 without breaking old code than I'd ever imagined.

There will still need to be some small amount of breaking changes, for which the 2.0 version number will be brought into play, but before I do that, 1.13 represents one last sweep to ensure that all my projects are not using "private APIs" on Yirgacheffe. As I mentioned in my previous weeknote, I've spent the last couple of weeks updating the LIFE pipeline for new functionality, and whilst I've been in there I've been catching up to the new 2.0 style API where I can: a job made significantly easier by the fact the import signature has been simplified from:

from yirgacheffe.layers import RasterLayer, VectorLayer
from yirgacheffe.window import Area
import yirgacheffe.operators as yo

to

import yirgacheffe as yg

Any file that doesn't have that single import is ready for tidying up and all those over imports will be broken in 2.0. They're actually now redirects in the later 1.x releases, as I restructured the code quite a bit this last half year, but I won't maintain those redirects when the 2.0 switch happens.

Then when I've found a place to update, the code again just mostly becomes simplified, particularly because some of the LIFE is a couple of years old and Yirgacheffe is now much more expressive. There's the simple bits like tidying APIs to be more like Pandas:

elevation_map = RasterLayer.layer_from_file("/some/path.tif")
range_map = VectorLayer.layer_from_file_like("/some/range.geojson", elevation_map)

That second argument on the vector layer load is a reference raster to set the map projection and pixel scale at which we need to rasterize the layer. However, Yirgacheffe can work all that out now because we changed how expressions in Yirgacheffe are materialised, allowing it to walk the AST and infer what if I combine these two data sources I should rasterize everything to match, which means this is now just:

elevation_map = yg.read_raster("/some/path.tif")
range_map = yg.read_shape("/some/range.geojson")

Much simpler and matching the read_csv and read_file APIs on other data-science libraries. However, even simpler yet is saving out results. Before we'd do something like:

layers = [elevation_map, range_map]
intersection = RasterLayer.find_intersection(layers)
for layer in layers:
    layer.set_window_for_intersection(intersection)

elevation_within_range = elevation_map * range_map
result = RasterLayer.empty_raster_layer_like(elevation_map, filename="/some/result.tif")
elevation_within_range.save(result)

This is now:

elevation_within_range = elevation_map * range_map
elevation_within_range.to_geotiff('/some/result.tif')

Again, because I restructured how calculations are materialised, no longer to I make it the user's job to work out if we should be intersecting or unioning data sources, I can infer that from the shape of the calculation. I can also just add a save method that avoids users creating their own result layer, and I know how big that should be because again I can inspect the calculation tree.

So for this trivial example we've about halved the amount of code, and made it look more like other data-science libraries Yirgacheffe's users are likely familiar with. Updating LIFE to use the new APIs was so pleasing as I managed to get rid of a lot of code that obfuscated from the scientific method encoded in these Python scripts.

---

So what's new in 1.13? Well, in several places Yirgacheffe lacked APIs for certain things, and so to get the job done I'd just used Yirgacheffe as far as I could, and then reached into layer objects and found the underlying GDAL object within, and used that directly. An example of that was saving multiband GeoTIFFs. As you can see in the above example, you call to_geotiff on an expression, so it can only contain one result (for now, I have plans on this for 3.0 :), and so we still had to call empty_Rater_layer_like and then write the data to the inner GDAL object directly. To fix this how you can do:

yg.to_geotiff("/some/multiband.tif", [calc1, calc2, calc3], labels=["birds", "mammals", "reptiles"])

This will generate a three band GeoTIFF and give each band a label that shows up in tools like QGIS.

The other thing I had to update was there was still once place where the default Yirgacheffe behaviour did the wrong thing when trying to guess the intersection or union of things based on operators. If you use constant values (e.g., add one to every value in this raster), it'd clip the constant size to the raster, rather than leaving the result global, and this tripped me up in LIFE and I'd still had to do set_window_for_union to fix it. I now fixed up the logic to handle that case better.

---

So nothing too exciting, but now that the LIFE pipeline is updated to the newer APIs and I've managed to slay the final examples of private API use, I think that means I can do one small update in the coming weeks that will nudge us up to a 2.0 by removing or changing a small number of APIs. It's funny to me that 2.0 will be a relatively small diff, and perhaps if I'm lucky actually a negative diff in line count, but that ignores all the hard work that went into turning this API ship around over the last six months whilst not breaking API compatibility.

The main changes will in fact be removing things like set_window_for_union and set_window_for_intersection which are stateful APIs that make some of Yirgacheffe's internals much messier than they'd otherwise need be. This calls are from a time before Yirgacheffe became a declarative library, and was just a way to do the map offset logic before you call read_array on a layer. Now Yirgacheffe can do even more for you without needing you to be explicit about things like this, and if I do need to add a way to modify how Yirgacheffe compiles expressions, I'll do it with in expression operators (similar to astype in both Numpy and Yirgacheffe), rather than do it via a stateful approach.

Exciting times, and as I say, I have some big ideas for Yirgacheffe 3.0, which I look forward to incrementally adding to 2.x over the next year :)