Hacker News: ryanworl

The Hitchhiker's Guide to Disaster Recovery and Multi-Region Kafka

ryanworl — Tue, 10 Jun 2025 15:22:25 +0000

Article URL: https://www.warpstream.com/blog/the-hitchhikers-guide-to-disaster-recovery-and-multi-region-kafka

Comments URL: https://news.ycombinator.com/item?id=44237969

Points: 5

# Comments: 0

New comment by ryanworl in "DuckLake is an integrated data lake and catalog format"

ryanworl — Tue, 27 May 2025 22:22:03 +0000

Does this trick preclude the ability to sort your data within a partition? You wouldn’t be able to rely on the row IDs being sequential anymore to be able to just refer to a prefix of them within a newly created file.

New comment by ryanworl in "ClickHouse gets lazier and faster: Introducing lazy materialization"

ryanworl — Tue, 22 Apr 2025 19:11:03 +0000

This is a well-known class of optimization and the literature term is “late materialization”. It is a large set of strategies including this one. Late materialization is about as old as column stores themselves.

Taking Out the Trash: Garbage Collection of Object Storage at Scale

ryanworl — Thu, 10 Apr 2025 18:35:45 +0000

Article URL: https://www.warpstream.com/blog/taking-out-the-trash-garbage-collection-of-object-storage-at-massive-scale

Comments URL: https://news.ycombinator.com/item?id=43646844

Points: 7

# Comments: 0

New comment by ryanworl in "The Query Condition Cache"

ryanworl — Mon, 31 Mar 2025 19:53:39 +0000

The equivalent to this feature is one of my favorite parts of Husky, Datadog’s storage and query system for event data.

https://youtu.be/mNneCaZewTg?si=N68fsBlYS3tuvLe3 begins at 34:32

Multiple Regions, Single Pane of Glass

ryanworl — Fri, 21 Jun 2024 15:48:59 +0000

Article URL: https://www.warpstream.com/blog/multiple-regions-single-pane-of-glass

Comments URL: https://news.ycombinator.com/item?id=40750782

Points: 20

# Comments: 3

New comment by ryanworl in "Tiered storage won't fix Kafka"

ryanworl — Mon, 29 Apr 2024 16:08:11 +0000

We're still drafting our next post in this series, but the answer is actually very simple: two tiers of object storage do not have the same drawbacks as a combination of object storage and local disk. We wanted to explain that in this post too, but it would've been unreasonably long.

We've designed WarpStream to work extremely well on the slower, harder-to-use one first, and that is how 95+% of our workloads run in production. The tiered storage solutions from other streaming vendors do the opposite, where they were first designed for local SSDs and then bolted on object storage later.

The equivalent would be if we were pitching our support for an even slower, cheaper tier of object storage like AWS S3 Glacier.

New comment by ryanworl in "Tiered storage won't fix Kafka"

ryanworl — Mon, 29 Apr 2024 15:54:46 +0000

This strategy will not work well for Apache Kafka because it is extremely IOPS hungry if you have more than a few partitions, and a replay of a large topic will require lots of IO bandwidth. It would work well e.g. a columnar database where a query targeting old data may only require reading a small fraction of the size of the volume, but Kafka is effectively a row-oriented storage system, so the IO pattern is different.

New comment by ryanworl in "Tiered storage won't fix Kafka"

ryanworl — Mon, 29 Apr 2024 15:31:22 +0000

(WarpStream co-founder here)

We're not talking about no disks as in no storage, just nothing other than object storage. This does have a latency trade-off, but with the advent of S3 Express One Zone and Azure's equivalent high-performance tier (with GCP surely not far behind), a system designed purely around object storage can now trade cost for latency where it makes sense. WarpStream already has support for writing to a quorum of S3 Express One Zone buckets to provide regional availability, so there's not an availability trade-off here either.

The Original Sin of Cloud Infrastructure

ryanworl — Thu, 14 Mar 2024 15:12:46 +0000

Article URL: https://www.warpstream.com/blog/the-original-sin-of-cloud-infrastructure

Comments URL: https://news.ycombinator.com/item?id=39705053

Points: 28

# Comments: 2

Kafka as a KV Store: deduplicating keys with just 128 MiB of RAM

ryanworl — Tue, 05 Mar 2024 16:00:29 +0000

Article URL: https://www.warpstream.com/blog/kafka-kv-store

Comments URL: https://news.ycombinator.com/item?id=39605247

Points: 16

# Comments: 0

New comment by ryanworl in "Announcing The Amazon S3 Express One Zone Storage Class"

ryanworl — Tue, 28 Nov 2023 19:30:32 +0000

Co-founder of WarpStream here. We wrote a short article explaining how the economics of this new storage class work for a system like ours: https://www.warpstream.com/blog/s3-express-is-all-you-need

S3 Express Is All You Need

ryanworl — Tue, 28 Nov 2023 19:04:44 +0000

Article URL: https://www.warpstream.com/blog/s3-express-is-all-you-need

Comments URL: https://news.ycombinator.com/item?id=38449827

Points: 163

# Comments: 85

New comment by ryanworl in "Unlocking Idempotency with Retroactive Tombstones"

ryanworl — Mon, 20 Nov 2023 17:16:48 +0000

WarpStream co-founder here. Implementing the Idempotent Producer feature for the Apache Kafka protocol was definitely a fun challenge for us. Please let me know if you have any questions!

New comment by ryanworl in "Minimizing S3 API costs with distributed MMAP"

ryanworl — Wed, 11 Oct 2023 17:01:59 +0000

[WarpStream co-founder]

WarpStream doesn't implement compacted topics today. It is on our roadmap, though. Compacted topics are typically not used in high-throughput workloads, so our plan is to delay compactions for longer than a disk-based system would to trade space amplification for write amplification.

New comment by ryanworl in "Kafka is dead, long live Kafka"

ryanworl — Tue, 08 Aug 2023 19:34:50 +0000

[WarpStream CTO here]

WarpStream flushes after 4MiB of data or a configurable amount of time. Flushes can also happen concurrently.

In general, we'd prefer to not introduce many knobs. We're running a realistic throughput testing workload in our staging and production environments 24/7, so we've configured most of the knobs already to reasonable defaults.

We just added support for other S3-compatible storage systems today: https://docs.warpstream.com/warpstream/reference/use-the-age...

New comment by ryanworl in "Kafka is dead, long live Kafka"

ryanworl — Tue, 08 Aug 2023 14:28:08 +0000

[WarpStream CTO here]

I'm not going to respond to your comment directly (we've already solved all the problems you've mentioned), but I thought I should mention for the sake of the other readers of this thread that you work for Redpanda which is a competitor of ours and didn't disclose that fact. Not a great look.

https://github.com/Lazin

New comment by ryanworl in "Kafka is dead, long live Kafka"

ryanworl — Mon, 07 Aug 2023 19:54:00 +0000

We're aiming for per-GB usage-based pricing that is significantly cheaper than the alternatives, but the BYOC model combined with our extremely efficient cloud control plane gives us a lot of flexibility here. That's why we can offer a free tier at all.

We're mostly just not sure yet, so your input would be appreciated.

New comment by ryanworl in "Kafka is dead, long live Kafka"

ryanworl — Mon, 07 Aug 2023 19:30:50 +0000

[WarpStream co-founder here]

Please sign up for our mailing list! We'll have some interesting things to announce related to this soon.

New comment by ryanworl in "Kafka is dead, long live Kafka"

ryanworl — Mon, 07 Aug 2023 19:18:20 +0000

With an appropriately configured client (i.e. one that retries and waits for requests to be acknowledged), another Agent would receive the retry and the event would be written to the topic at that time.