I think I'm going to go with 8x RAIDz2 VDEVs 10x HDDs each, so that the 20 drives in the internal drive enclosure could be 2 separate VDEVs and not mix with the 60 in the external enclosure.

For software, at least with MinIO it's possible to do rolling updates/restarts since the 5 instances in docker-compose are enough for proper write quorum even with any single instance down.

The link is a 10G 9K MTU connection, the server is only accessed via that local link.

Essentially, the drives being HDD are the only real bottleneck (besides the obvious single-node scenario).

At the moment, all writes are buffered into the NVMes via OpenCAS write-through cache, so the writes are very snappy and are pretty much ingested at the rate I can throw data at it. But the read/delete operations require at least a metadata read, and due to the very high number of small (most even empty) objects they take a lot more time than I would like.

I'm willing to sacrifice the write-through cache benefits (the write performance is actually an overkill for my use case), in order to make it a little more balanced for better List/Read/DeleteObject operations performance.

On paper, most "real" writes will be sequential data, so writing that directly to the HDDs should be fine, while metadata write operations will be handled exclusively by the flash storage, thus also taking care of the empty/small objects problem.

Basically, I have a single big server with 80 high-capacity HDDs and 4 high-endurance NVMes, and it's the S3 endpoint that gets a lot of writes.

So yes, for now my best candidate is ZFS + Garage, this way I can get away with using replica=1 and rely on ZFS RAIDz for data safety, and the NVMEs can get sliced and diced to act as the fast metadata store for Garage, the "special" device/small records store for the ZFS, the ZIL/SLOG device and so on.

Currently it's a bit of a Frankenstein's monster: using XFS+OpenCAS as the backing storage for an old version of MinIO (containerized to run as 5 instances), I'm looking to replace it with a simpler design and hopefully get a better performance.

Recently I've been looking into Garage and liking the idea of it, but it seems to have a very different design (no EC).

Lately I'm running this like once every 2-3 weeks and it's still surprisingly nice to use. The problems only really manifest when you never run this "maintenance" snippet and only use the `brew install`, which will attempt to do the housekeeping stuff when you want/expect it the least.

  brew update && brew upgrade && brew autoremove && brew cleanup && brew doctor

If you don't want the certificate to be in the CT logs, your only options are a private CA or things like CF Origin certificate, depending on how the domain is intended to be accessed.

It's not the end user that "needs" CT, it is a mechanism to ensure no shady CA can misissue a certificate without being caught. Requirements like that are written in blood (see Symantec).

I am uninstalling Firefox after 16 years of sustained everyday use in dual-browser split setup alongside Safari. I will also make sure everyone I know still using FF gets the note.

I am going back to Chromium-based browsers that are openly "sell-your-data" from day 1 and I am willing to accept the Google monopoly if it means that dung beetles like Firefox are not able to prey on loyal supporters by bait-and-switching the ToS whenever they feel like. Hopefully their telemetry shows them a good picture of how their decisions affect their market share.

Hell, my 2011 issue about wrong handling of ::first-letter pseudoclass on bugzilla hasn't got a single meaningful update since, but I was willing to sacrifice the standards compliance for the promise of "privacy". There is no more privacy.