I'm not gonna lie, though, you're hitting a sore point. I'm absolutely kicking myself for not putting SQLite under test when I joined. The first few projects I put under test after joining were all projects that had a few properties: 1. The project had a path to a commercial deal 2. The project likely had interesting bugs to find

TBH, I didn't expect that one of the most widely used and well-tested pieces of software in existence would have a bug our fuzzer could find in 15 mins. (To be clear, this is not a hit on SQLite, just that our fuzzer is stupid good at finding race conditions. SQLite remains one of my favorite pieces of software ever.)

With this in mind, are there other similar projects that I'm not thinking of that I should put under test? Before you ask, I'm already scaling up my SQLite harness to cover more of its state space and run against master. Maybe I'll find something novel in SQLite after all! Never too late to check.

https://news.ycombinator.com/item?id=49278424 https://news.ycombinator.com/item?id=49278521

Also appreciate the nice words at the end :) I'm feeling a bit ganged up on.

Which is: "Given the knowledge of a kind of bug that can happen in one Raft implementation, can a similar workload find similar bugs in other Raft implementations?" Turns out it can.

The workload I put in place does exactly that[1]. It runs a write workload from multiple processes concurrently on the same SQLite database to cause writes to build up in the WAL, and runs checkpoints concurrently. This exercises the portion of the WAL code that, from a trivial read-through, is most likely to contain bugs (and turns out, did!).

Said differently, this is exactly the approach we take with all stateful transactional systems. I am only sad that I didn't do this experiment months earlier, as it would have saved Tailscale and the SQLite team a lot of time.

[1]: https://github.com/antithesishq/sqlite/blob/3.51.2-instrumen...

My goal, which I believe I accomplished, was to determine if Antithesis would have found this bug if we had put the SQLite WAL under test.

[1]: https://github.com/antithesishq/antithesis-skills

I'm only sad that I didn't put SQLite under test earlier in the year, or I would have found this issue right away. If you look at the workload[1], you can see how simple it is. Exactly the same kind of workload we write every day to help our customers find bugs.

If you have any questions about our process or how debugging with Antithesis works, please let me know! Thanks for reading!

[1]: https://github.com/antithesishq/sqlite/blob/3.51.2-instrumen...

Comments URL: https://news.ycombinator.com/item?id=48466036

Points: 2

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=48414365

Points: 6

# Comments: 4

Let's say a maintainer of an open-source project decided they hated Linux and added a check that, when run on Linux, the project would run `rm -rf `. Would you trust that person not to do other unreasonable things in the future?

SingleStore (f.k.a. MemSQL) used lock-free skiplists extensively as the backing storage of their rowstore tables and indexes. Adam Prout (ex CTO) wrote about it here: https://www.singlestore.com/blog/what-is-skiplist-why-skipli...

When SingleStore added a Columnar storage option (LSM tree), L0 was simply a rowstore table. Since rowstore was already a highly optimized, durable, and large-scale storage engine, it allowed L0 to absorb a highly concurrent transactional write workload. This capability was a key part of SingleStore's ability to handle HTAP workloads. If you want to learn more, take a look at this paper which documents the entire system end-to-end: https://dl.acm.org/doi/10.1145/3514221.3526055

I do like the B-tree aware grouping idea. This seems like a useful optimization for larger scan-style workloads. It helps eliminate the need to vacuum as much.

Have you considered doing other kinds of optimizations? Empty pages, free pages, etc.