It's definitely still a lot more than the zero work required for Google Photos, but I feel like even even a semi-technical user can test an Immich backup without a ton of hassle. Start a new/empty immich instance in an empty directory, and follow the wizard for restoring a backup (and copy of your photos folder) taken from the non-empty instance.

But you’re generally right. I think that’s why most databases have the notion of a WAL, which is carefully append-only. But the non-WAL data files in most DBs I’ve used are accessed via mmap.

1. Writes try to compress away duplicate metadata at the application layer, which causes them to issue scattered writes when new metadata shows up.

2. Indexing is also surprisingly log-line/application-layer aware, such that index writes might also be scattering.

3. The indexes themselves seem like they could benefit from an append-mostly write model with periodic compaction rather than a mutate-in-place model.

4. I was surprised that the journal’s “WAL” doesn’t seem to be a major concern of a lot of the code. For a database, supporting reads “through” the WAL with periodic application back to the data files (“checkpoints” in RDBMS) seems like something I’d expect to see more of here. But I don’t really have deep understanding of the code, so I may be missing that it’s doing that already.

The choice of mmap instead of regular file writes here isn’t, as others have proposed, a design flaw. I think that makes sense given what journald is (a database) and how significant its durability concerns are. And it looks like the code does spend a lot of time trying to be careful about which blocks/pages are dirtied. But this is a famously hard-to-get-write (ha!) area so perhaps defects are present at that layer.

The systemd developers are talented in their area; I am not a systemd hater. However, “talented at low-level OS design” is not the same as “talented at building a database from scratch”, and I think that shows here.

I strongly feel like this system could be a wrapper around SQLite, which is definitely something that could be integrated everywhere journald is used (license-wise and compatibility-wise). I’m puzzled as to why that wasn’t chosen as an approach: a SQLite vfs implementation that handled compression and online rotation seems like it would have resulted in a design that’s both more interoperable and less prone to flaws like this one.

I also think that a per-log-emitter setting that doesn’t eagerly persist to disk (wait for page cache flush) would be very useful to have available—perhaps even as a default—for user-level/init6 level logs that are OK with a potential for data loss on kernel panic.

You said:

> When something isn't packaged for these users but they feel like it's popular/useful enough to be, they'll open a request with their distribution for someone to maintain that

I don’t think that happens nearly often or quickly enough for most users to be satisfied. Hence why so much software is released through tertiary PPAs or .deb download links and so on.

Comments URL: https://news.ycombinator.com/item?id=49288644

Points: 39

# Comments: 43

And yet insane quantities of programs used by millions are only available via raw .deb file downloads, or random alternative PPAs, or via programming language package managers.

The distro maintainer system is impressive, and I’m thankful for it. But saying “it has worked for lots of packages” says nothing about the many it has not worked for.

Distro maintainer curation is a godsend for security and reliability. Distro maintainers and people who work with them to get things packaged are heroes. But let’s not pretend that maintainer curation is something the average user trying to run a new piece of software cares about.

Online backups/replicas are nice until you’re charged for network traffic or have to recreate the replica from scratch, at which point the initial-restore-then-hook-up-the-WAL-stream dance is prone to all sorts of racy issues. If you’re lucky enough to have only a single process talking to the DB, SQLite seems like a nice way to sidestep that complexity while keeping a simple backup story.

Heck, this is basically the Redis model: a single process/thread coordinates all access to the data, and occasionally forks off a background job to snapshot the state somewhere. From that perspective, the Tailscale controller binary is a database; SQLite is just the data file format.