Comments URL: https://news.ycombinator.com/item?id=48480561

Points: 2

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=48334445

Points: 3

# Comments: 0

[Disclosure: I used to work at StarTree. Still a fan!]

Comments URL: https://news.ycombinator.com/item?id=47577186

Points: 8

# Comments: 8

The agents themselves are not likely going to be doing the high cardinality queries or they will keel over. They have limited memory buffers. They will take many seconds to return results. They are likely going to be limited in terms of QPS.

From the blog: > Apache Iceberg, with data stored as Parquet on S3, and most of the system implemented in Go

You have just ensured that queries will have a p99 >1 second. This is kind of antithetical to having an agent be fast.

You couldn't run any sort of real-time service, where hundreds of thousands to millions of events were occurring per second, and you needed to adjust to that in milliseconds.

The terms "p99" and "QPS" do not occur anywhere in the article. Which leaves the question of scalability to a user's imagination.

I applaud the direction. I am looking for objective evidence.

It's somewhat different from how x86 chips do simultaneous multithreading (SMT),

https://www.redpanda.com/blog/nvidia-vera-cpu-performance-be...

Comments URL: https://news.ycombinator.com/item?id=47405385

Points: 2

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=47064457

Points: 1

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=46620570

Points: 2

# Comments: 0

kevstev wrote just above about Kafka being written to run on spinning disks (HDDs), while Redpanda was written to take advantage of the latest hardware (local NVMe SSDs). He has some great insights.

As well, Apache Kafka was written in Java, back in an era when you were weren't quite sure what operating system you might be running on. For example, when Azure first launched they had a Windows NT-based system called Windows Azure. Most everyone else had already decided to roll Linux. Microsoft refused to budge on Linux until 2014, and didn't release its own Azure Linux until 2020.

Once everyone decided to roll Linux, the "write once run everywhere" promise of Java was obviated. But because you were still locked into a Java Virtual Machine (JVM) your application couldn't optimize itself to the underlying hardware and operating system you were running on.

Redpanda, for example, is written in C++ on top of the Seastar framework (seastar.io). The same framework at the heart of ScyllaDB. This engine is a thread-per-core shared-nothing architecture that allows Redpanda to optimize performance for hardware utilization in ways that a Java app can only dream of. CPU utilization, memory usage, IO throughput. It's all just better performance on Redpanda.

It means that you're actually getting better utility out of the servers you deploy. Less wasted / fallow CPU cycles — so better price-performance. Faster writes. Lower p99 latencies. It's just... better.

Now, I am biased. I work at Redpanda now. But I've been a big fan of Kafka since 2015. I am still bullish on data streaming. I just think that Apache Kafka, as a Java-based platform, needs some serious rearchitecture,

Even Confluent doesn't use vanilla Kafka. They rewrote their own engine, Kora. They claim it is 10x faster. Or 30x faster. Depending on what you're measuring.

1. https://www.confluent.io/confluent-cloud/kora/

2. https://www.confluent.io/blog/10x-apache-kafka-elasticity/

1. https://www.cio.com/article/4084855/ibm-to-cut-thousands-of-...

2. https://www.newsobserver.com/news/business/article312796900....

And here is a picture of Ben Lorica 罗瑞卡 interviewing Jay Kreps and other industry leaders at The Hive back on the evening of 25 February 2015. I believe they were talking about strategies for implementing Lambda Architecture.

All of which is to say: I have been a big fan of both companies for a long, long time. While today I am at employed at Redpanda Data, a direct competitor of Confluent, I hope to set aside any "team"-based bias to provide a sober and honest appraisal.

First, IBM has been shrinking. They were at 345,000 employees as of their 2020 Annual Report. But the COVID-19 pandemic was only one of many setbacks the company faced when Arvind Krishna took the helm as CEO. By December 2024 the employee base shrank to 270,000 — a drop of nearly 22%.

IBM revenue in 2020: $73.6B.

IBM revenue in 2024: $62.75B — a less-precipitous drop of 15%.

Revenue per employee over that period rose from $213k to $232k.

Confluent on its own? $400k.

And to compare: Amazon earns $580k per employee. Microsoft generates over $1M per. Nvidia? $4M-$5M.

And now, in November, they announced thousands of more layoffs. No one seems safe, regardless of job title. Those cut include positions in "artificial intelligence, marketing, software engineering and cloud technology."

Next, IBM has had a mixed record as a steward of acquisitions. Red Hat has doubled in revenues since their 2019 acquisition. For a while its headcount continued to grow, as much as 19,000 by 2023. But then it was forced into layoffs by parent IBM in April of that year, and then each year since, even while it remains one of the highest margin businesses in their portfolio.

SoftLayer — "IBM Cloud Classic" — also suffered significant layoffs in early 2025, with offshoring sending jobs to India.

DataStax had layoffs in 2023-2024, even before its acquisition was announced. Maybe they were "trimming the fat" to get into a shape to be acquired.

As a person with a long career in marketing, I know that many of the first roles to be jettisoned at a newly-acquired company tend to be in go-to-market organizations. Sales, Marketing, Developer Relations, Documentation, Training, Community, Customer Service. These tend to be seen as "nice to haves" by upper management. But their loss guts organizations and hollows out user-facing teams and open source communities.

My hope is that Confluent is spared as much of the pain and turmoil as possible. That, like Red Hat, it is run autonomously as much as possible.

[Crossposted from LinkedIn here, where you can see the photo mentioned: https://www.linkedin.com/feed/update/urn:li:activity:7404052...]