someone should inform the orange POTUS of this: https://www.architecturaldigest.com/story/did-donald-trump-j...

My impression is this is a modern civilization stupidity in general.

apropos Dead Kennedys track: https://www.youtube.com/watch?v=lVULKYbAeEo

as is ext4

and that seems like something btrfs should fix at some point

You linked copy_file_atomic_at_full(), why? That function is not in the normal rotation path for journald, it's only used in a workaround when clearing FS_NOCOW_FL fails.

Rotation does not rewrite the log file normally, but there is a hole-punching operation though for reclaiming unused space.

https://github.com/systemd/systemd/blob/main/docs/JOURNAL_FI...

There was mailing list discussion at the time journald was conceived though, you can find it if you look.

https://0pointer.de/blog/projects/the-journal.html might be a good entry-point.

the last time I contributed to journald upstream was to fix a degenerate behavior with many journal files: https://github.com/systemd/systemd/commit/176f73272e6e3116ca...

that makes a dramatic difference for those hitting this case, but it only gets things from nearly unusable to slow-as-usual.

But I never really made any effort to change the on-disk structure or how writes were performed. My focus was more on the read performance for journalctl and stability of the daemon.

Back when I was paid to fix things in journald at CoreOS ages ago, it couldn't even avoid getting killed by its own service watchdog.

My impression back then was the on-disk format dispersed the information too much within the same file, and those individual datums being written at discontiguous offsets were quite small, far smaller than an IO block size or even a disk sector size.

Seemed like a write amplification problem due to the file format. If you write a few bytes into some arbitrary position within a file, the storage has to write back the whole block, despite your only changing a tiny fraction of it. If those few bytes happened to cross a block boundary, guess what? two blocks get written.

The format had no consideration for these block-oriented storage details, then doing the IO via mmap rubs salt into the wound since the kernel has to try guess what to prefetch asynchronously... but I don't think that aspect amplifies the writes above what plain buffered IO would do - maybe I'm wrong. I'd expect the mmap aspect to be causing more/mispredicted reads, and polluting the page cache with unrelated contents (you tend to end up with the entire journal cached IIRC, if you have enough memory). I suppose there's probably compounding of the write amplification problem since the kernel will be dirtying pages at page size granularity vs. 512b sectors, and you have the same issue of small writes landing on page boundaries dirtying two pages. So that aspect of using mmap for the writes probably is exacerbating the problem.

  > Intel developed QPI at its Massachusetts Microprocessor Design Center (MMDC) by
  > members of what had been the Alpha Development Group, which Intel had acquired
  > from Compaq and HP and in turn originally came from Digital Equipment
  > Corporation (DEC).[8] Its development had been reported as early as 2004

QPI was Intel's answer to the competition AMD's Athlon / HyperTransport had created

Of course we're fucked.

that's quite the optimistic end! There's absolutely zero chance this ends with a pickup powered by the venerable Toyota 22R-E I4 or an equivalent.

"I4, proven" proven to be genai slop and nothing more.

which upon writing I realized may not even be something folks born after 2010 have even heard of...

https://en.wikipedia.org/wiki/Window_shopping