Queues are great when you have a unit of work that needs to be completed, retried, and tracked individually. For example, a shop might need to call a payment processor API that can fail or timeout, and retry it until it succeeds, while polling on the frontend for the state of that particular message. With a queue, you can insert a message tracking that payment, and have a queue processor that keeps getting sent it until it succeeds or has failed too many times.

In a queue each item is its own thing that's important to someone, and queues give you APIs to interact with that particular item.

K2 is for moving large volumes of data around. Pricing is per GB, not per message. Records are produced and consumed in bulk, and what matters is that all records are processed, but no one is querying the state of a particular record. K2 also supports multiple consumers for the same record, and long term retention. For example, all of your applications may emit events when things happen, and those events need to be read by an alerting system, a system that durably stores them, and a system that uses them to build ML features.

We went with a simpler and more user friendly consume API, that also allows much higher levels of read parallelism (particularly important if you're using something like Workers, which parallelize well but aren't very powerful individually).

But we know many companies are invested in the Kafka ecosystem, and we want to provide an easy on (and if necessary, off) ramp for them.

[0] https://developers.cloudflare.com/basin-pipelines/

> I do wonder if we will see an expansion of the s3 api to support more of these use cases

This is actually an area where I think we have a big leg up on folks building on top of S3. My team (which built K2) sits next to the R2 team, and we have the opportunity to co-evolve the products in mutually beneficial ways.

Compared to self-hosted or cloud-hosted Kafka (e.g., Amazon MSK or Confluent), K2 is much cheaper, and fully serverless. There are no clusters to manage or scale, and consistent performance even as you vastly increase the amount of data.

The main downside is produce (and end to end) latency is higher (around 1s p99) than systems that rely on local disk replication, like Kafka.

So it's great if you're trying to move a huge amount of data around, or for use cases where cost is more important than latency.

Streaming always felt like it was about to happen, and it kept not happening, in a year of the linux desktop sort of way. I (and others, who also optimistically started stream processing startups) thought it was a technology problem but ultimately it's a demand problem: very few companies actually need low latency processing. And continuously running, stateful systems are inherently more complex to operate and evolve compared to batch systems.

Now that relatively low-latency ingest into warehouses and datalakes is easily achieved, it's really hard to make the argument to invest in complicated streaming systems.

That said, it still feels like we've landed in a suboptimal spot. Stream processors (in particular, those following the dataflow model) solve two problems that are hard in batch systems:

1. Determining and signaling completeness for a time period (when have I likely received all of the data for 12:01pm, such that I can now safely process it)

2. Avoiding expensive recomputations for periodic queries

I do think long term we'll see some amount of streaming influence moving back into the batch systems, particularly as object storage gives us the ability to run stateful workloads with less operational headache.

(If anyone else finds these problems interesting, I'm hiring for my stream processing team at Cloudflare. Email in my profile)

ABC started very simple, because most of the performance information for folk music isn't written down, it's inferred by the player according to the idiom of that particular tradition. As usage of ABC has grown it's gotten more powerful but still falls far short of formats designed for western classical, like MusicXML or Lilypond.

[0] https://abcnotation.com/history

There's also an element of violin-specific pitch detection; if you play violin for long enough, you can recognize the specific timbre of different notes on a violin (particularly easy for open strings) which helps ground you while listening to a tune.

(Disclaimer: I work for Cloudflare)