Why using Slack is bad for GeoTIFF Rasters

In the past I’ve written about how GeoTIFF is one of the most common datas formats used for geospatial data. A GeoTIFF is just a standard TIFF image with a few extra fields on it that mean geospatial tools such as QGIS or libraries such as GDAL know what geographic region this data is referring to, and what map projection system its stored relative to. For example, here is a GeoTIFF I generated showing me where in Sweden you might find moose (meese?):

Perhaps I’m sufficiently excited by the notion of where the moose can be found that I want to share this with one of my colleagues via our workplace Slack channel. So I drop the file in, they download it, but then they complain that the image is now all wrong:

If we zoom out a bit, we find not only does the image look funny, it’s no longer in the right place! I couldn’t really make a screenshot of this, as the moose data has been rendered so small that by the time I zoom out enough that you see the coast of Africa it’s just a couple of pixels wide on my screen:

If look closely I can see that QGIS is warning me something is up with this layer that has been sent via Slack:

So what’s happened here? This is particularly mysterious as in the past my colleagues and I have successfully sent GeoTIFFs through slack with no ill effect.

---

Let’s take a look with some command line tools at the TIFF itself. If I use gdalinfo to query the original file, I see what I’d expect:

$ gdalinfo smb.tif
Driver: GTiff/GeoTIFF
Files: smb.tif
Size is 2867, 3060
Coordinate System is:
GEOGCRS["WGS 84",
	ENSEMBLE["World Geodetic System 1984 ensemble",
		MEMBER["World Geodetic System 1984 (Transit)"],
		MEMBER["World Geodetic System 1984 (G730)"],
		MEMBER["World Geodetic System 1984 (G873)"],
		MEMBER["World Geodetic System 1984 (G1150)"],
		MEMBER["World Geodetic System 1984 (G1674)"],
		MEMBER["World Geodetic System 1984 (G1762)"],
		MEMBER["World Geodetic System 1984 (G2139)"],
		ELLIPSOID["WGS 84",6378137,298.257223563,
			LENGTHUNIT["metre",1]],
		ENSEMBLEACCURACY[2.0]],
	PRIMEM["Greenwich",0,
		ANGLEUNIT["degree",0.0174532925199433]],
	CS[ellipsoidal,2],
		AXIS["geodetic latitude (Lat)",north,
			ORDER[1],
			ANGLEUNIT["degree",0.0174532925199433]],
		AXIS["geodetic longitude (Lon)",east,
			ORDER[2],
			ANGLEUNIT["degree",0.0174532925199433]],
	USAGE[
		SCOPE["Horizontal component of 3D system."],
		AREA["World."],
		BBOX[-90,-180,90,180]],
	ID["EPSG",4326]]
Data axis to CRS axis mapping: 2,1
Origin = (11.026820112567126,69.106496492030672)
Pixel Size = (0.004491576420598,-0.004491869987684)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  (  11.0268201,  69.1064965) ( 11d 1'36.55"E, 69d 6'23.39"N)
Lower Left  (  11.0268201,  55.3613743) ( 11d 1'36.55"E, 55d21'40.95"N)
Upper Right (  23.9041697,  69.1064965) ( 23d54'15.01"E, 69d 6'23.39"N)
Lower Right (  23.9041697,  55.3613743) ( 23d54'15.01"E, 55d21'40.95"N)
Center      (  17.4654949,  62.2339354) ( 17d27'55.78"E, 62d14' 2.17"N)
Band 1 Block=2867x2 Type=Byte, ColorInterp=Gray

But if we do the same on the file that went via Slack we see:

$ gdalinfo ~/Downloads/smb.tif
Driver: GTiff/GeoTIFF
Files: /Users/michael/Downloads/smb.tif
Size is 2867, 3060
Image Structure Metadata:
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0, 3060.0)
Upper Right ( 2867.0,    0.0)
Lower Right ( 2867.0, 3060.0)
Center      ( 1433.5, 1530.0)
Band 1 Block=2867x352 Type=Byte, ColorInterp=Gray

GDAL doesn’t really know anything about the file. Why this is so becomes more apparent if we just look at the TIFF file metadata in both. GeoTIFFs are just regular TIFF files with a few special header tags. You can see this with tiffinfo from the command line on the original file:

$ tiffinfo smb.tif
TIFFReadDirectory: Warning, Unknown field with tag 33550 (0x830e) encountered.
TIFFReadDirectory: Warning, Unknown field with tag 33922 (0x8482) encountered.
TIFFReadDirectory: Warning, Unknown field with tag 34735 (0x87af) encountered.
TIFFReadDirectory: Warning, Unknown field with tag 34736 (0x87b0) encountered.
TIFFReadDirectory: Warning, Unknown field with tag 34737 (0x87b1) encountered.
=== TIFF directory 0 ===
TIFF Directory at offset 0x8 (8)
  Image Width: 2867 Image Length: 3060
  Bits/Sample: 8
  Sample Format: unsigned integer
  Compression Scheme: LZW
  Photometric Interpretation: min-is-black
  Samples/Pixel: 1
  Rows/Strip: 2
  Planar Configuration: single image plane
  Tag 33550: 0.004492,0.004492,0.000000
  Tag 33922: 0.000000,0.000000,0.000000,11.026820,69.106496,0.000000
  Tag 34735: 1,1,0,7,1024,0,1,2,1025,0,1,1,2048,0,1,4326,2049,34737,7,0,2054,0,1,9102,2057,34736,1,1,2059,34736,1,0
  Tag 34736: 298.257224,6378137.000000
  Tag 34737: WGS 84|
  Predictor: none 1 (0x1)

All those “Unknown field with tag…” warnings are the GeoTIFF extensions that tiffinfo doesn’t understand. And if we again look at the file that we downloaded from Slack:

$ tiffinfo ~/Downloads/smb.tif
=== TIFF directory 0 ===
TIFF Directory at offset 0x85dda4 (8773028)
  Image Width: 2867 Image Length: 3060
  Bits/Sample: 8
  Compression Scheme: None
  Photometric Interpretation: min-is-black
  FillOrder: msb-to-lsb
  Orientation: row 0 top, col 0 lhs
  Samples/Pixel: 1
  Rows/Strip: 352
  Planar Configuration: single image plane
  Page Number: 0-1

All those GeoTIFF tags are gone, and what was a GeoTIFF has now just become a regular old TIFF after its journey through Slack.

---

What was particularly puzzling about this is that I know for a fact I’ve sent GeoTIFFs through Slack before without hitting this issue. And when I tried to reproduce the issue I found I couldn’t - it just seemed to be some particular GeoTIFFs it didn’t like. I tried many things to narrow it down: was it some particular tags it didn’t like? was it certain map projections? was it file size? was it data type? For each idea I was able to find a counter example where it worked. As I tested this the Slack channel where you can talk to yourself (which I finally had a use for!) filled up with GeoTIFFs:

Eventually I realised that the issue is related to when Slack decides to generate a preview: if there is a preview, the downloaded file has no metadata, whereas is there isn’t a preview you get the original file!

Now, on one hand this isn’t really progress, as I still don’t know what settings on the GeoTIFF cause Slack to render a review versus not - I have the same problem and just a quicker way to diagnose it perhaps. But then on the other hand at least I can diagnose it now: if I send a colleague a GeoTIFF and it renders a preview, it is now ruined. I assume that if you have a preview and you download the image again then you get a new image from the original data rather than the one you uploaded. Or perhaps, as my colleague Patrick pointed out, once Slack decides to treat it like an image it strips out metadata for security reasons - but then if that’s the case its poor that I can get images through that don’t get cleaned up like that.

It does remind me a bit of the early days of mobile networks, where your mobile operator would rewrite images you downloaded on their servers to be more highly compressed, so as to save them bandwidth. You hear less of that these days, and I have checked the actual image data, and I’ve not yet found one where the data itself was changed, just the metadata.

---

Slack fiddling in this way me cost the better part of days work, because we didn’t spot that it had mangled one of our datasets and so we had confused results, and so it’s particularly vexing that I can’t find a root cause, but at least I have a work around now: I’ve taken to putting any GeoTIFFs I send to colleagues into a zip file now - not to save space (given the GeoTIFFs are compressed putting them in a zip often makes them slightly larger), but rather to stop Slack fiddling with them. Not the best, but if I do it as a matter of course, I never have to think about this again, at least until Slack decide to fiddle with zip files.