ELT-Bench is another recent example. It was the first serious attempt at a benchmark for data engineering workloads, published about a year ago.

A few days ago, a follow-up paper from a group that includes one of the original authors audited the benchmark itself. The team gfound that the benchmark has structural issues that biased results.

Here’s the paper: https://arxiv.org/abs/2603.29399

None of these are new though, the industry has gone through all that before just in a smaller scale and there’s a lot to learn from that. Here’s a post I wrote on the parallels we see today to what happened with the benchmarketing wars of the database systems.

https://www.typedef.ai/blog/from-benchmarketing-to-benchmaxx...

I was so surprised when I tried baked sprouts for the first time (use a really host cast iron skilet for even better results) that I started to believe that every vegetable can be delicious as long as you bake it!

I think that a much cleaner messaging on what this tool is for would help.

Also a question about the implementation, why DuckDB for a CRM?

Something like SQLite feels like a much natural fit for a CRM where you primarily create, update and maybe delete records and you really care for the integrity of the data model.

From a quick look on the data model, everything seems to be a VARCHAR, if this is the case, why not just store everything in the file system instead? You do that with the md files and whatever is getting extracted from the SaaS tools.

Claude Code et.al. are good examples of that. Diffs, user approval flows, non-linear flows in general and a ton of text buffered are all elements that we know really well how to handle in web interfaces but are challenging for the terminal.

Lance[1] (the format, not just LanceDB) is a great example, where you have columnar storage optimized for both analytical operations and vector workloads together with built-in versioning for dataset iteration.

Plus (very important) random access, which is important for stuff like sampling and efficient filtering during curation but also for working with multimodal data, e.g. videos.

Lance is not alone, vortex[2] is another one, nimble[3] from Meta yet another one and I might be missing a few more.

[1] https://github.com/lance-format/lance [2] https://vortex.dev [3] https://github.com/facebookincubator/nimble

I'm curious why you think I'm acting like it's all or nothing. What I was trying to communicate is the exact opposite, that it's not all or nothing. Maybe it's the way I articulate things, I'm genuinely interested what makes it sound like this.

What I said in my original comment is that AI delivers when it's used by experts, in this case there was someone who was definitely not a C compiler expert, what would happen if there was a real expert doing this?

When the systems turn into something trivial to manage with the new tooling, humans build more complex or add more layers on the existing systems.

This result reminded me of the C compiler case that Anthropic posted recently. Sure, agents wrote the code for hours but there was a human there giving them directions, scoping the problem, finding the test suites needed for the agentic loops to actually work etc etc. In general making sure the output actually works and that it's a story worth sharing with others.

The "AI replaces humans in X" narrative is primarily a tool for driving attention and funding. It works great for creating impressions and building brand value but also does a disservice to the actual researchers, engineers and humans in general, who do the hard work of problem formulation, validation and at the end, solving the problem using another tool in their toolbox.

Comments URL: https://news.ycombinator.com/item?id=46008137

Points: 6

# Comments: 0

There are other formats though that it can be compared to.

The Lance columnar format is one: https://github.com/lancedb/lancedb

And Nimble from Meta is another: https://github.com/facebookincubator/nimble

Parquet is so core to data infra and widespread, that removing it from its throne is a really really hard task.

The people behind these projects that are willing to try and do this, have my total respect.

Comments URL: https://news.ycombinator.com/item?id=45983042

Points: 3

# Comments: 1

Comments URL: https://news.ycombinator.com/item?id=45714625

Points: 2

# Comments: 0