Diom includes implementations for common backend primitives such as cache, key-value, idempotency, rate-limiting, queues, and streams, with more on the way.

While building Svix, we had to reimplement the same backend primitives that everyone have to reimplement. We also constantly felt the tension between building something custom on top of existing infra (like Redis and Postgres) and adding more dedicated services (like RabbitMQ and Kafka) which we would then need to configure, operate, back up, and maintain. This was even worse for us because Svix is open-source, so additional infrastructure meant additional burden on our customers.

Six months ago we finally decided to build Diom, and focus on developer experience and ease of operation. It's open source, self-contained, and manages its own storage using fjall (a fast LSM-tree-based storage similar to RocksDB). It requires no external runtime dependencies (no redis/postgres/kafka/etc), and supports running as a single node or a highly-available Raft based cluster.

The goal of Diam is to provide developers with the backend primitives they need without having to write custom code on top of Redis, RabbitMQ, Kafka, or even need to run them at all. It currently supports cache, key-value, idempotency, rate-limiting, queues, and streams. We also plan on adding auth-tokens, distributed settings, feature flags, and other common components; as well as adding more functionality to existing components.

Diom favors ease of operation over scale, so it doesn't match Kafka-level throughput or very high QPS like Redis and Dragonfly. However, most products and developers don't process multiple terabytes and billions of events per second anyway. That said, Diom can still hit high performance for its target use-cases as it implements higher-level primitives rather than basic operations. Additionally, because the primitives live in the same process as the storage, there are fewer network round-trips, which keeps latency low.

It uses HTTP/2 with msgpack as the wire protocol (works fine from browsers), and ships a CLI and SDKs for Python, TypeScript, Rust, Go, and Java, with more on the way.

We have Svix fully ported to Diom and continuously running tests and simulated workloads in one of our staging environments. GA (general availability) is planned for later this year, once we've moved Svix production workloads over.

Repo (MIT licensed): https://github.com/svix/diom

Docs: https://docs.diom.com

Live playground: https://diom.com/playground

I'm excited to finally share Diom, and would love to hear what everyone thinks, and what other components you would like us to build! Would also love help figuring out what to call this. We currently say "component platform," but I'm not a fan of the name.

Comments URL: https://news.ycombinator.com/item?id=48120719

Points: 4

# Comments: 0

Diom is a single Rust binary that replaces Redis, RabbitMQ, Kafka, and a lot of custom code for many common use-cases. It's open-source, self-contained, and has no external runtime dependencies.

While building Svix, we had to reimplement the same backend primitives that everyone reimplements when building a service: robust caching, rate-limiting, idempotency, and more. We also constantly felt the tension between building something custom on top of existing infra (like Redis and Postgres) and adding more dedicated services (like RabbitMQ and Kafka) which we would then need to configure, operate, back up, and maintain. This was even worse for us because Svix is open-source, so additional infrastructure meant additional burden on our customers.

Six months ago we finally decided to build Diom. It's a self-contained open-source service that people can run in their infrastructure with a focus on developer experience and ease of operation. It manages its own storage using fjall (a fast LSM-tree-based storage similar to RocksDB) and therefore requires no external runtime dependencies (no redis/postgres/kafka/etc), and supports running as a single node or a highly-available Raft based cluster.

The goal is for Diom to replace Redis, RabbitMQ, Kafka, and a lot of custom code for most use-cases, by offering pre-built implementations for common backend primitives. It currently supports cache, key-value, idempotency, rate-limiting, queues, and streams. We also plan on adding auth-tokens, distributed settings, feature flags, and other common components; as well as adding more functionality to existing components.

The closest existing project is probably NATS with JetStream, which is also a single self-contained binary with KV, streams, and queues. One of the main differences is that Diom ships higher-level primitives like idempotency, rate limiting, and cache as first-class APIs, rather than something you build on top of KV and streams.

Diom favors ease of operation over scale, so it doesn't match Kafka-level throughput or very high QPS like Redis and Dragonfly. However, most products and developers don't process multiple terabytes and billions of events per second anyway. That said, Diom can still hit high performance for its target use-cases as it implements higher-level primitives rather than basic operations. Additionally, because the primitives live in the same process as the storage, there are fewer network round-trips, which keeps latency low.

It uses HTTP/2 with msgpack as the wire protocol (works fine from browsers), and ships a CLI and SDKs for Python, TypeScript, Rust, Go, and Java, with more on the way.

We have Svix fully ported to Diom and continuously running tests and simulated workloads in one of our staging environments. GA (general availability) is planned for later this year, once we've moved Svix production workloads over.

Repo (MIT licensed): https://github.com/svix/diom

Docs: https://docs.diom.com

Live playground: https://diom.com/playground

I'm excited to finally share Diom, and would love to hear what everyone thinks, if you find it useful, and what other components we should build! Would also love help figuring out what to call this. We currently say "component platform," but I'm not a fan of the name.

Comments URL: https://news.ycombinator.com/item?id=47996356

Points: 5

# Comments: 0

As for the $40m number: I also saw this number, but I don't think it's correct. E.g. Germany recently bought 140 of them for $165m. Ref: https://www.i24news.tv/en/news/israel/defense/1649255166-ger...

For perspective: Patriot missiles cost $4m each.

As for traveling to the future: that sounds like fun!

So you support many more than you realize!

Edit: changed "perfect" to "improve", as I meant "perfect" as "betterment" not in terms of absolute perfection.

I wonder how long it'll take them writing a compositor from scratch.

Now, if what you're saying is that with super highly optimized sections of a codebase, or extremely specific circumstances (some kernel drivers) you'd need a bit of unsafe rust: then sure. Though all of a sudden you flipped the script, and the unsafe becomes the exception, not the rule; and you can keep those pieces of code contained. Similarly to how C programmers use inline assembly in some scenarios.

Funny enough, this is similar to something that Rust did the opposite of C, and is much better for it: immutable by default (let mut vs. const in C) and non-nullable by default (and even being able to define something as non-null). Flipping the script so that GOOD is default and BAD is rare was a huge win.

I definitely don't think Rust is a silver bullet, though I'd definitely say it's at least a silver alloy bullet. At least when it comes to the above topics.

I think the idea of developers developing a "bugs antenna" is good in theory, though in practice the kernel, Redis, and many other projects suffer from these classes of bugs consistently. Additionally, that's why people use linters and code formatters even though developers can develop a sensitivity to coding conventions (in fact, these tools used to be unpopular in C-land). Trusting humans develop sensibility is just not enough.

Specifically, about the concurrency: Redis is (mostly) single-threaded, and I guess that's at least in part because of the difficulty of building safe, fast and highly-concurrent C applications (please correct me if I'm wrong).

Can people write safer C (e.g. by using sds.c and the likes)? For sure! Though we've been writing C for 50+ years at this point, at some point "people can just do X" is no longer a valid argument. As while we could, in fact we don't.

Comments URL: https://news.ycombinator.com/item?id=46466119

Points: 3

# Comments: 0

Slightly different as I generate a PGP key on the computer and then load it to the Yubikey, which means I can have backup keys with the same secret keys.

I never really got "touch to use" working though, if anyone knows how to do it with GPG keys I'd really appreciate it!