Zebras All the Way Down - Bryan Cantrill, Uptime 2017

On this topic (and related in many ways) is Bryan Cantrill's talk Zebras All the way Down.[1] The gist is that in medicine there's the concept of diseases that are zebras and horses, rare and common, and the saying goes "when you hear hoof beats think horses not Zebras". Unlike medicine, in software most the easy problems have already been abstracted away or made trivial through our software stacks, so often what's left are Zebras. What used to be the exceedingly rare bugs now account for a large proportion of the odd behavior we see in very robust tech stacks, because the easy bugs have been hunted down and eliminated already. 1: [this video]

In my time writing databases, I found that not only the mount-flags could result in data-loss despite fsync, given you use fsync as it is supposed to be used [0], but even enterprise-grade hardware would at times (at that scale [1]) drop and/or corrupt a few bits here and there. That reminds me of the case where AWS famously revealed that a single NIC in their ginaromous S3 fleet flipped a single bit once in a while and that caused an outage because their gossip-daemon responsible for fleet health-checks failed spectacularly [2]. Bryan Cantrill's talk on the realted topic of hardware/firmware bugs is pretty good [3]. [0] [link] [1] [link] [2] [link] [3] [this video]

(Sheepishly) My only reference is a talk by Bryan Cantrill[1] where he states that an exec at a hard-drive manufacturer stated it was ".8 nanometers". I will try to find a better source than that. EDIT: There is a paper from 2016 which did an analysis in the difference in the flying height of the head during different operations, and it was measured in Angstroms[2]. I couldn't find one that actually gives a precise value of the flying height. There is a 2011 paper which states that some system they were testing allowed for 4-9 nm flying heights[3] which is about half-an-order-of-magnitude larger than the claim -- but that's already 7 years old. [1]: [this video] [2]: [link] [3]: [link]

> When we really should have been formally addressing problems from first principles. Because if our OS can't recover gracefully when the disk fills up, then we don't really have anything. And so many OSs (including our desktops) panic when the disk fills up, that we don't really have anything. At a low enough level, all of the tools we rely upon are trash. Unfortunately, the OS can't really "recover" most of the time because it is being constantly lied to by several(!!!) layers of uncooperative hardware/firmware. It's zebras all the way down: [this video] > In this case, by letting programs wait until the user has cleared off disk space If your system is really in a no-disk-space-left state you might not be able to login to fix it: [link] And even assuming you can login, sometimes you can't delete files either; you need to free space in another way before being able to delete files. [link] Filesystems and persistence are hard.

> Or you can spend half a billion dollars to solve the issue in hardware. And hope that your hardware/firmware doesn't ever get bugs. Or you can do checksumming at the hardware layer and checksumming at the software/FS layer. Protection in depth. ZFS has caught issues from hardware, like when LBA 123 is requested but LBA 456 is delivered: the hardware-level checksum for LBA 456 was fine, and so it was passed up the stack, but it wasn't actually the data that was asked for. See Bryan Cantrill's talk "Zebras All the way Down": * [this video] And if checksums are not needed for a particular use-case, make them toggleable: even ZFS has a set checksums=off option. My problem is not having the option at all .