He also more or less conceded Adrian’s framing that we still haven’t had a real “PageRank moment for intelligence” yet even while defending Transformers as the strongest thing that currently works and scales on the current hardware.

One of the sharpest lines in the whole debate is probably Llion’s version of the local-minimum argument: Kaiser may be right up until the day a real breakthrough arrives and then wrong forever.

There was a back-and-forth on scaling, hardware constraints, continual learning and latent reasoning.

Comments URL: https://news.ycombinator.com/item?id=48210458

Points: 5

# Comments: 3

TL;DR Both formats have their perks: Apache Iceberg offers an open, flexible architecture with surprisingly fast query performance in some cases, while Databricks Delta Lake provides a tightly managed, all-in-one experience where most of the operational overhead is handled for you.

Setup & Methodology

We used the TPC-H 1 TB dataset which is a dataset of about 8.66 billion rows across 8 tables to compare the two stacks end-to-end: ingestion and analytics.

For the Iceberg setup:

We ingested data from PostgreSQL into Apache Iceberg tables on S3, orchestrated through OLake’s high-throughput CDC pipeline using AWS Glue as catalog and EMR Spark for query.. Ingestion used 32 parallel threads with chunked, resumable snapshots, ensuring high throughput. On the query side, we tuned Spark similarly to Databricks (raised shuffle partitions to 128 and disabled vectorised reads due to Arrow buffer issues).

For the Databricks Delta Lake setup: Data was loaded via the JDBC connector from PostgreSQL into Delta tables in 200k-row batches. Databricks’ managed runtime automatically applied file compaction and optimized writes. Queries were run using the same 22 TPC-H analytics queries for a fair comparison.

This setup made sure we were comparing both ingestion performance and analytical query performance under realistic, production-style workloads.

What We Found

We used OLake to ingest to Iceberg and was about 2x faster - 12 hours vs 25.7 hours on Databricks thanks to parallel chunked ingestion.

Iceberg ran the full TPC-H suite 18% faster than Databricks.

Cost: Infra cost was 61% lower on Iceberg + OLake (around $21.95 vs $50.71 for the same run).

here are the overall result and our ideology on this-

Databricks still wins on ease-of-use: you just click and go. Cluster setup, Spark tuning, and governance are all handled automatically. That’s great for teams that want a managed ecosystem and don’t want to deal with infrastructure.

But if your team is comfortable managing a Glue/AWS stack and handling a bit more complexity, Iceberg + OLake’s open architecture wins on pure numbers faster at scale, lower cost, and full engine flexibility (Spark, Trino, Flink) without vendor lock-in.

read our article to know more on our steps followed and the overall benchmarks and the numbers around it curious to know what you people think ofcourse these are numbers but it largely depends on your experience too of how you adopted in your org

Comments URL: https://news.ycombinator.com/item?id=45976740

Points: 9

# Comments: 2

What stood out to me wasn’t just the raw numbers but how it’s being used. The Claude for Chrome extension basically lets the model act right inside your browser it clicks, types, navigates sites, fills spreadsheets, etc. It’s not just about generating text or code, it feels better and easier to work with an assistant that can actually do the work for you.

What I like about this approach is that it doesn’t feel limited to engineers. Sure, devs can use it to speed up tasks, but non-technical folks could benefit just as much since it works as a simple browser extension. While other models are chasing raw speed and size, this one seems aimed at practical usability.

Comments URL: https://news.ycombinator.com/item?id=45432035

Points: 2

# Comments: 1

As part of that journey, we’ve been contributing upstream to the Apache Iceberg Go ecosystem. this week, our PR to enable writing into partitioned tables got merged (https://github.com/apache/iceberg-go/pull/524)

However that may sound niche, but it unlocks a very practical path for Go services to write straight to Iceberg (no Spark/Flink detour) and be query-ready in Trino/Spark/DuckDB right away.

what we added : partitioned fan-out writer that splits data into multiple partitions, with each partition having its own rolling data writer efficient Parquet flush/roll as the target file size is reached, all the usual Iceberg transforms supported: identity, bucket, truncate, year/month/day/hour Arrow-based write for stable memory & fast columnar handling

and why we’re bullish on Go for building our platform - OLake?

the runtime’s concurrency model makes it straightforward to coordinate partition writers, batching, and backpressure. small static binaries → easy to ship edge and sidecar ingestors. great ops story (observability, profiling, and sane resource usage) which is a big deal when you’re replicating at high rates. where this helps right now: building micro-ingestors that stream changes from DBs to Iceberg in Go. edge or on-prem capture where you don’t want a big JVM stack. teams that want cleaner tables (fewer tiny files) without a separate compaction job for every write path.

For data teams still worried about Go, we have our case study helps you : check the benchmarks we’re hitting thanks to the language’s lightweight model See numbers here: https://olake.io/docs/benchmarks

If you’re experimenting with Go + Iceberg, we’d love to collaborate as we believe in open source :)

repo: https://github.com/datazip-inc/olake/

Comments URL: https://news.ycombinator.com/item?id=45413064

Points: 2

# Comments: 1