I read both versions and agree that the second half of the first version was very abstract and difficult to follow. While I would consider the first half of the new version more edited than rewritten, the second half got a significant overhaul which fixed almost all of my issues with it and made for (in my opinion) a much more satisfying ending. I would recommend giving the new version another chance, though those who read the first version may find the new character names distracting. (Most didn't bother me, but Marion Wheeler -> Marie Quinn never felt quite right.)

> Note on the name: SCIP is pronounced the same way as “skip” and it’s a recursive acronym that stands for “SCIP Code Intelligence Protocol.”

> SCIP is also a purposeful nod to SICP (Structure and Interpretation of Programs), a book about analyzing programs.

https://sourcegraph.com/blog/announcing-scip#:~:text=SCIP%20...

For example, cp /path/to/dataset/mountpoint/.zfs/snapshot//path/to/file ~/path/to/file

Depending on if you're being optimistic or pessimistic, either 1) neither I nor ZFS did anything wrong or 2) both ZFS and I did some things wrong. Either way, neither one nor the other is particularly at fault.

While I said "ZFS lost its mind" for title length reasons, it really would be more accurate to say "ZFS appeared to loose its mind", since as I learned, everything makes sense when you consider the encryption root.

My only disagreement is that the only possible improvement is in documentation. I answered this in a reply to OpenZFS developer robn in another thread (https://old.reddit.com/r/zfs/comments/1ntwrjx/mind_the_encry...) which I'll copy here:

> This is great writeup, and I really appreciate you taking the time on it. With my OpenZFS dev hat on, it's often quite difficult to understand exactly how people are using the things we make, especially when they go wrong - what were they expecting, what conceptual errors were involved, and so on. I'm passing it around at the moment and will give it a much slower and more thoughtful read as soon as I can. Thanks!

> While it's fresh on your mind, what would be one simple change that we could make today that would have prevented this is or made it much less likely? Doc change, warning output, etc. I have some ideas, but I don't want to lead the witness :)

First off, thank you for taking the effort to try and understand this from the OpenZFS side. It's really easy to dismiss this kind of thing as user error (which is true) since OpenZFS did actually behave as designed (which is also true), rather than taking it as an opportunity to better understand and improve the user experience.

When I think of the factors that lead me to make the mistake of not sending a snapshot of the encryption root, I think it comes down to a difference in expectations vs reality. When I think of a snapshot, I conceptualize it as fully consistent version of a dataset at a point in time (which as far as I know is still true for unencrypted datasets). Native encryption violates that expectation by 1) storing the wrapping key parameters in the encryption root which may be a different dataset and therefore exists outside of the snapshot and 2) allowing the wrapping key and dataset master keys to get out of sync.

If I send a snapshot from one pool to another, I expect ZFS to send everything necessary to reproduce the same state on the destination pool. As an uneducated user, I'd find it very unintuitive that I also need to send a new snapshot of another empty, unchanged dataset which through some "spooky action at a distance" affects the decryptability other child encrypted datasets just because it's the parent dataset at the root of the encrypted tree. I expect datasets to be isolated from one another. Users shouldn't have to know about wrapping keys and master keys, let alone worry about keeping them in sync across multiple datasets.

While I do think the docs could be improved to emphasize the importance of the encryption root (especially in zfs-send -w, --raw which doesn't even mention it) which would've made debugging the issue a bit easier, I'm not sure how much that would've helped prevent the issue in the first place. The reality we must face is that people don't read the docs unprompted to challenge their fundamental expectations; they work with the mental model they have and only consult the docs when they have a question to answer.

What I do think would've really helped is if zfs-recv could check the wrapping key parameters on the encryption root when it sees a new encrypted master key in the send stream and abort if they don't match. This wouldn't prevent every scenario, (eg. if instead of forgetting to send a snapshot of the encryption root, you forgot to send snapshots of the child encrypted datasets), but it would have prevented this one and would be a step in the right direction.

In the long term, I'd really love for OpenZFS to treat keeping wrapping keys and master keys in sync as an invariant that is always maintained so users don't need to know about encryption roots, wrapping keys, or master keys and can ignore them like any other implementation details. I've only just begun thinking about some potential options in the solution space, but I have a feeling this will not be an easy problem to fully solve. I'd love to hear your ideas as an actual OpenZFS developer.

I emphatically agree, unencrypted ZFS on top of GELI or LUKS encrypted block devices is the way to go for now. Plus it also has the benefit of not leaking metadata like a sieve.

My mistake was placing too much trust in ZFS's reputation for data integrity; clearly not all features hold that value in the same regard.

The openness of OpenZFS was a real saving grace. If this had occurred on a propriety SAN, that data would be gone forever.

Comments URL: https://news.ycombinator.com/item?id=45410930

Points: 6

# Comments: 3