On of the things that Sweden is known for is its allemansrätten law, which gives people access to the land - what we'd call the "right to roam" in the UK. I feel this is a good label for my lidar work: can we create a place where people can explore their country virtually using already open data that is just locked away from them.

I've shared a couple of little views on my weeknotes (here and here) of the Swedish lidar data from Lantmäteriet, both limited to around 200,000 data points. This limitation isn't from what the browser can do, as it'll happily do one to two orders of magnitude more, but rather just the volume of data. Locally I was able to load an entire single tile of the Lantmäteriet lidar scan, covering an area of 2.5km per side, which was around 20 million points, but that file was around 200 MB even when compressed.

So my achievement last week was getting it so that I could load not just one tile, but a 3x3 grid of tiles (around 194 million points in total, 7.5 km per side), but way more important: I also changed it so that anyone can access it by changing how the data is stored to let it be loaded hierarchically. This means anyone can now explore this little corner of Sweden for themselves.

In addition to more lidar data, I've also changed now the data is presented, so the false colour height map isn't the only option any more: you can now see the lidar data catagories in the original source data, and you can see it mapped to the land cover classification using the same labelling and colour scheme from the underlying data set (which is a way of explaining why the labels are in Swedish :).

On top of that I've also expanded the example tree detection data to cover more areas so that it's more interesting to see what the algorithms do, including obvious errors:

I've also included the camera locations from SVT's The Great Moose Migration as another data layer (called DSÄ kameror, Den Stora Älgvandringen being the Swedish title of the show). This year's Great Moose Migration should begin on the 22nd of April, and was in part the inspiration for this work. I realised that despite watching it on TV for over five years now, when I started looking at 3D visualisations of the land data, I saw the area in a whole new light: I could see why the meese might choose to swim in one area rather than another because I could actually see the distances involved. On TV they use these amazing zoom lenses to cover a wide area, but high zoom lenses compress distance in a way that makes it very hard to appreciate what the distances involved are, but on this map I can see clearly where the river crossing is narrower and wider near the cameras.

I've also made some usability tweaks. The main one being that as you move around the location is stored in the URL so you can share what you've found with other people. I really think the power of this kind of viewer is allowing people from other domains to prod at the data and start asking questions about what is there, and to do that they need a way to both record what they're seeing for themselves and share it with others.

---

So how did I get it to jump two orders of magnitude in points I could share with people? To achieve the significant increase in data volume I switched from using Point Cloud Data format and moved to Cloud Optimised Point Cloud. COPC is a specialisation of the common LAS or LAZ file formats, which is actually the format the original data from Lantmäteriet was provided in (that is, they use LAZ format, not COPC). COPC organises the data internally wihtin the LAZ file in a specific hierarchy so that you can work out how to quickly access data that gives you a quick overview at a zoomed out level without having to read the entire file, and then similar you can zoom in to areas and the points are clustered within the file in such a way that you can get just that area out. Just this change alone did most of the work, after that it's been trying to tweak how much data to load when. I feel this still needs quite a bit of work, as the algorithm it's using now is quite clunky, but it's a start, and the hard part of working out if we can even do this has been done.

One unexpected side effect of moving to COPC: because similar points are closer and so have similar data properties, they compress better: just going to COPC got that original tile of 15M points down from a 600 MB LAZ file, which I'd then turned into a 200 MB PCD file, down to 105 MB - which is pretty impressive that we get close to a byte per point!

On the client side we're loading the data both on a tile by tile basis and then heirarchically within the times as we adjust the level of detail based on the zoom, and throwing away data we can no longer see or need at a particular zoom level. Again, I've not attempted to make any of this very efficient yet, but I think it shows one of the trade-offs in building this kind of system. In theory for this example I could just load all 9 tiles into one COPC laz file and just use its internal heirarchy to work out what to load, but if my aim is to scale this up to a country wide result, I can't put all of Sweden in a single file (reasonably), so there has to be some point at which the data is tiled into multiple files, and I suspect also a point where you have different files with different zoom levels entirely. Anil pointed me at some relevant work in this domain, so I have some homework to do before I try tackle this side of things.

1. Swedish does have a plural for moose (en älg, flera älgar), and I’m with the half of the community of moose migration watchers that thinks meese (one moose, many meese) is a valid one for English :)

   ↩︎︎

2. Technically a reservoir according to the Global Lakes and Wetlands Database we’ve been using in STAR

   ↩︎︎