[0] https://news.ycombinator.com/item?id=45223827

We developed OpenZL initially for our own consumption at Meta. More recently we've been putting a lot of effort into making this a usable tool for people who, you know, didn't develop OpenZL. Your feedback is welcome!

- A white paper: https://arxiv.org/abs/2510.03203

- A blog post: https://engineering.fb.com/2025/10/06/developer-tools/openzl...

- The documentation website: https://openzl.org/

Comments URL: https://news.ycombinator.com/item?id=45494592

Points: 33

# Comments: 4

Code: https://github.com/facebook/openzl

Documentation: https://openzl.org/

White Paper: https://arxiv.org/abs/2510.03203

The first stage of Zstd does LZ77 matching, which transforms the input into "sequences", a series of instructions each of which describes some literals and one match. The literals component of the instruction says "the next L bytes of the message are these L bytes". The match component says "the next M bytes of the input are the M bytes N bytes ago".

If you want to construct a match between two strings that differ by one character, rather than saying "the next N bytes are the N bytes M bytes ago except for this one byte here which is X instead", Zstd just breaks it up into two sequences, the first part of the match, and then a single literal byte describing the changed byte, and then the rest of the match, which is described as being at offset 0. The encoding rules for Zstd define offset 0 to mean "the previously used match offset". This isn't as powerful as a Levenshtein edit, but it's a reasonable approximation.

The big advantage of this approach is that it doesn't require much additional machinery on the encoder or decoder, and thus remains very fast. Whereas implementing a whole edit description state machine would (I think) slow down decompression and especially compression enormously.

[0] https://datatracker.ietf.org/doc/html/rfc8878#name-repeat-of...

Ultimately, Zstd is a byte-oriented compressor that doesn't understand the semantics of the data it compresses. Improvements are certainly possible if you can recognize and separate that framing to recover a contiguous view of the underlying data.

[0] https://github.com/facebook/zstd/blob/v1.5.7/lib/compress/zs...

(I am one of the maintainers of Zstd.)

I get exactly that green cast and muted color range off of my flatbed scans (Epson v800). This is a really intriguing path to fixing them I hadn't considered.

It seems like the writeup here doesn't specify what you're using for the actual imaging? A flatbed scanner? A camera?

By far the more common case for image stabilization is one in which the photographer is hand-holding the camera and may not frame the subject until the moment before the exposure begins. The camera movement will likely be several orders of magnitude (~4 to 7) larger than the drift that you want to measure. A low pass filter will tell you nothing at all.

At a certain point we can just start using guide stars [0].

[0] https://en.wikipedia.org/wiki/Guide_star

Sure, humans act against their own interests all the time. Sometimes we do so for considered reasons, even. But that's because humans are messy and our interests are self-contradictory, incoherent, and have a fairly weak grip on our actions. We are always picking some values to serve and in doing so violating other values.

A strongly and coherently aligned AI would not (could not!) behave that way.

Whereas Arithmetic encoding is actually practical, extensively used, and a direct analogue to the stick.