There were also a bunch of fun things with quirks around unicode filename handling which made me sad (that was just a matter of testing against a ton of clients).

As for CalDAV and CardDAV - as others have said, JMAP Calendars/Contacts will make building clients a lot easier eventually... but yeah. My implementation of syncing as a client now is to look for sync-collection and fall back to collecting etags to know which URLs to fetch. Either way, sync-collection ALSO gives a set of URLs and then I multi-get those in batches; meaning both the primary and fallback codepath revert to the multi-get (or even individual GETs).

The downsides of developing and testing this stuff as we were writing it up!

We've finished rewriting the objectid generation to give smaller sized and more sortable IDs (they're an inverse of nanosecond internaldates now, plus some extra magic on the low bits for IMAP appends to avoid clashes)... which we wanted to speed up and reduce disk usage for the offline mode.

Next up is indeed updating to the latest spec on calendars and contacts. Files might take a bit longer, I really want to do some desktop clients for the files system, we have a really nice files backend at Fastmail which is only accessible via our interface or WebDAV right now.

UIDs don't change, but of course they can be deleted so it's a gappy list, meaning you can request even quite a large looking range of UIDs and get nothing back.

Message numbers change in every session, and also change every time you get an EXPUNGE. They're basically an ordered list without gaps, so you do a memmove at the offset of the EXPUNGE each time you get an expunge.

There are efforts like UIDONLY (RFC9586) to avoid having to keep that mapping at all, and there's OBJECTID (RFC8474) to let you cache a lot more even when UIDs are changed or when messages are moved between folders.

Comments URL: https://news.ycombinator.com/item?id=45453135

Points: 166

# Comments: 111

(am a member of this select club)

[TIL - it's not even as old as me!] https://australianfoodtimeline.com.au/1978-launch-of-big-m/

You can see all the code for that in a handful of merge requests in the public cyrus-imapd repository on github at https://github.com/cyrusimap/cyrus-imapd/

Over the past few weeks, I've been helping out with the last bits of code modification, largely the changes on https://github.com/cyrusimap/cyrus-imapd/pull/5539 if you're interested.

This morning we rolled out a build which we'd tested extensively on our staging and staff servers, but missed that for older v19 mailboxes which hadn't been upgraded to v20, the code to check if messages belonged in a thread incorrectly marked them all as missing.

This made MOST emails appear missing for most customers, clearly a very bad situation.

We immediately rolled back, but in the hurry missed that an unrelated change to correct subject matching for some languages (Japanese users had reported the issue, but possibly others as well) had changed the thread version, so new threads then had failed reads (making some, though many fewer, messages appear blank in the UI). There were about 50 million attempts to read those values over 15,000 users, because our UI was keeping on retrying thinking it was just a temporary synchronisation issue because the previous request told it there was a Thread to fetch data for. Ouch. https://github.com/cyrusimap/cyrus-imapd/pull/5527 contains those changes.

Anyway, since the only difference between the old and new records was normalisation of subjects, I wrote a tiny patch to let the old code read the newer records and just deployed that, which made all the emails re-appear for everyone again. This is the one bit of code from all this which isn't in a public repo, but it's two lines of: if (version == 2) version = 1;

Meanwhile, the real bug is fixed https://github.com/cyrusimap/cyrus-imapd/pull/5553 And a test has been written to prove it: https://github.com/cyrusimap/cyrus-imapd/pull/5554

But we'll wait until Monday to upgrade again, when we have fresh eyes available to watch that it's OK.

...

P.S. this is almost entirely unrelated to the UI changes. The underlying reason we're doing these changes IS related to UI changes, it's there to make offline mode use storage more efficiently on your device because the IDs are smaller and provide better data locality, but the timing is purely coincidental. The Cyrus changes have been done almost exclusively by the team in the USA and the UI changes by the team in Australia, and our deploy timelines were not synchronised.

The larger context is that we're making a major change to how we create IDs for email and mailboxes over the JMAP protocol. The old IDs are a UUID for mailboxId and the first 25 chars of the sha1 of the message for the emailId, prefixed by an 'M'. The new IDs are the createdmodseq for the mailbox prefixed by a 'P' (these are pretty short for most users) and a reverse counter of nanoseconds of the message internaldate (delivery time) for the emailId. This gives good storage density for offline and good data locality in databases for the email listings.

You can see all the code for that in a handful of merge requests in the public cyrus-imapd repository on github at https://github.com/cyrusimap/cyrus-imapd/

Over the past few weeks, I've been helping out with the last bits of code modification, largely the changes on https://github.com/cyrusimap/cyrus-imapd/pull/5539 if you're interested.

This morning we rolled out a build which we'd tested extensively on our staging and staff servers, but missed that for older v19 mailboxes which hadn't been upgraded to v20, the code to check if messages belonged in a thread incorrectly marked them all as missing.

This made MOST emails appear missing for most customers, clearly a very bad situation.

We immediately rolled back, but in the hurry missed that an unrelated change to correct subject matching for some languages (Japanese users had reported the issue, but possibly others as well) had changed the thread version, so new threads then had failed reads (making some, though many fewer, messages appear blank in the UI). There were about 50 million attempts to read those values over 15,000 users, because our UI was keeping on retrying thinking it was just a temporary synchronisation issue because the previous request told it there was a Thread to fetch data for. Ouch. https://github.com/cyrusimap/cyrus-imapd/pull/5527 contains those changes.

Anyway, since the only difference between the old and new records was normalisation of subjects, I wrote a tiny patch to let the old code read the newer records and just deployed that, which made all the emails re-appear for everyone again. This is the one bit of code from all this which isn't in a public repo, but it's two lines of: if (version == 2) version = 1;

Meanwhile, the real bug is fixed https://github.com/cyrusimap/cyrus-imapd/pull/5553 And a test has been written to prove it: https://github.com/cyrusimap/cyrus-imapd/pull/5554

But we'll wait until Monday to upgrade again, when we have fresh eyes available to watch that it's OK.

...

P.S. this is almost entirely unrelated to the UI changes. The underlying reason we're doing these changes IS related to UI changes, it's there to make offline mode use storage more efficiently on your device because the IDs are smaller and provide better data locality, but the timing is purely coincidental. The Cyrus changes have been done almost exclusively by the team in the USA and the UI changes by the team in Australia, and our deploy timelines were not synchronised.

So you'd have something like:

DKIM2: i=1; mf=sender@trusted.com; rt=accounts-payable@example.com; d=trusted.com

DKIM2: i=2; mf=bounce@example.com; rt=me@mydomain.com; d=example.com

So I could tell that the message came through example.com, and verify their signature on the message, as well as verify that trusted.com had intended the message to go to example.com in the previous hop.

https://datatracker.ietf.org/doc/draft-chuang-dkim2-dns/

https://datatracker.ietf.org/doc/draft-gondwana-dkim2-header...

https://datatracker.ietf.org/doc/draft-gondwana-dkim2-modifi...

https://datatracker.ietf.org/doc/draft-robinson-dkim2-bounce...

https://datatracker.ietf.org/doc/draft-robinson-dkim2-messag...

https://datatracker.ietf.org/doc/draft-gondwana-dkim2-motiva...

Note that it doesn't help avoid Google actually sending out a message with user-provided text in it, but it does stop it being replayed to you without Google intending it, because the SMTP FROM/TO are protected.

The motivation draft doesn't include technical detail, see early drafts of the technical detail in the various related docs at:

https://datatracker.ietf.org/wg/dkim/documents/