(Disclaimer: I work for Cloudflare)

> Look, I am sorry. But if you go to Jump Trading and Jane Street and say “hello, I have an unregulated poorly designed mechanism that could lead to $50 billion of market value collapsing overnight, would you like to trade with me,” they are going to say yes, but their eyes are going to light up, you know? If at Time 0 you give them an extremely gameable system that can produce billions of dollars of profit, at Time 10 your system is going to be a smoking wreckage and they are going to have billions of dollars of profit. That’s their whole job, you know? I couldn’t tell you in advance what all the intermediate steps will be, and in fact in hindsight I cannot tell you what the intermediate steps actually were, how Jump and Jane Street made money off the collapse of Terra. But as a heuristic, I mean, come on. Terra was like “hello we have a balloon full of money, here is a pin, dooooooon’t pop the balloon.” Guess what!

This isn't new either, people have been building OLAP storage engines into MySQL/Postgres for years, e.g., pg_ducklake and timescale.

It's San Francisco/Austin/Seattle/New York City/London/Lisbon

Hey HN, I'm hiring engineers for the new Cloudflare Data Platform (https://blog.cloudflare.com/cloudflare-data-platform). We do streaming ingest and processing, managed Iceberg catalogs, and a distributed query engine—all built on R2 and the Cloudflare Edge. We're making it ridiculously easy for Cloudflare customers to build data lakes.

I'm looking for Rust engineers with experience working on databases, stream processing engines, and query engines. You'll get to work on incredibly cool tech on a huge scale.

Note: we are hiring for on-site roles in one of our engineering hubs, not hiring remote at this time.

If you're interested, shoot me an email (in profile) with subject HN and a brief description of your background and interest in the role.

I found it hard to get next to reliably treat static content as actually static (and thus cacheable), and it felt like a huge bundle of complexity for such a simple use case.

In a typical company you'll have jobs, some scheduled, some ad-hoc, at a range of sizes. Most of them won't be cost-effective to run on a GPU instance, so you need a scheduling layer that estimates the size of the job and routes it to the appropriate hardware. But now what if the job is too big to run on your GPU machine? Now we either have to scale up our GPU cluster or retry it on our more flexible CPU cluster.

And this all assumes that your jobs can be transparently run across different executors from a correctness and performance standpoint.

There are niches where this makes sense (we run the same 100TB job every day and we need to speed it up), as well and large and sophisticated internal infra teams that can manage a heterogenous cluster + scheduling systems, but it's not mass-market.

(I have the same workflow, via Obsidian)