Inspired by the latest release notes on one of the group's other projects, I spent a moment looking at the load times of Yirgacheffe, my Python geospatial library. I'd started to notice that compared to many other Python packages I use Yirgacheffe was relatively slow to import. Just a few seconds, which isn't much in the context of a geospatial pipeline, but I feel creates a poor impression when you're manually loading it in an interactive session. It felt particularly vexing when in my mind there was no reason for it: Yirgacheffe shouldn't be doing anything interesting until the user asks it to, so I wasn't sure why it wasn't a near instant load.

I had a look around at how I might dig into this, and came across Tuna. Tuna is a profiling tool for Python that has a specific mode for investigating import issues like the one I was interested in. Once installed via pip, to profile an import is as simple as:

$ python -X importtime -c "import yirgacheffe" 2> import.log
$ tuna import.log

The first command does the profiling, and the second command fires up a web server with an interactive flame graph that shows where all the time is spent:

In this instance it let me quickly see that the main culprit was pytorch, taking over half the import time. This is somewhat silly, as pytorch is used for one operator (conv2d) and only for the CPU backend, the GPU backend using MLX instead. I suspect I could remove pytorch and just use MLX, as it does have a CPU backend also, but for now I just simply moved the pytorch import from the top of a Python file and made it a local import in the conv2d function on the CPU backend. One line relocated, and Yirgacheffe's import time was more than halved!

The second most heavy import was pyproj, which is a Python wrapper over the PROJ coordinate transform library. This one was a little more invasive, as I use pyproj in quite a few places and it's almost always going to be needed. Rather than find all the places I was using it and work out a bunch of local import changes I used another trick here, which is to use lazy-loader, a Python package that is designed to let you wrap packages in a lazy loading function, so that the rest of your code is unchanged, you just change the import from:

import pyproj

to

import lazy_loader as lazy
pyproj = lazy.load('pyproj')

After that your code is the same, but the loading of pyproj wont happen until you first try and execute anything from that package.

I have to confess, I expected improving the Yirgacheffe import time to be a bit more of a slog, but thankfully there are excellent tools out there that make it a breeze to keep on top of your imports. Within twenty minutes or so I had got Yirgacheffe's import time down to a third of its former value, and fast enough I was happy to leave it there for now, but I know what to do if I need improve it more in the future.