NYC Tech Talk Series: How Google Backs Up the Internet

The "petabyte for a century" problem statement he mentions near the top is fun: can you preserve 1 PB with better-than-even odds of all the bits coming back right in a century? He wrote something about how he saw the durability situation in 2010: [link] . He seems to define the problem to allow for maintenance, e.g. check-and-repair operations are discussed. Broadening a little, I read it as "if you're trying to be more cautious than you usually want for commercial/academic online storage or even backup, what do you do? And would it work?" A lot (but not all) of the author's Queue piece talks about stats from online storage, which doesn't have some wins you can get if you're entirely about long-term durability. In online systems, heavy ECC seems out of favor compared to replication for performance reasons, but additional LDPC or RS at the app layer can absorb a substantial % of your volumes having problems, or a ton of random bad blocks. (In a more near-line context, Backblaze uses RS: [link] ) Same for tape -- offline LTO is slow and drives are costly, but the cost/TB and the rated lifespan seem like advantages over HDDs for this specific goal, at least in a narrow engineering sense. A pile of cryptographic hashes that fits on one device can help you check for and localize errors without sending everything over a network/doing the full ECC dance. If the hash being broken and data tampered with is in your threat model (hey, weird things can happen in a century), you can also