(Disclaimer: I work at DuckDB Labs)

I updated your reference [0] with this information.

https://archive.fosdem.org/2025/schedule/event/fosdem-2025-5...

- c8gd.4xlarge - this has a single 950 GB NVMe SSD.

- c5ad.4xlarge - this has 2 x 300 GB disks, which I put in a RAID 0 array. There are no c6ad.4xlarge instances, so this is the closes NVMe-enabled approximate to ClickBench's most popular choice, c6a.4xlarge.

I also added results from my local dev machine, a MacBook M1 Max with 64 GB RAM and 10 cores.

Here are the results:

  | machine        | cold_run_avg | cold_run_sum | hot_run_avg | hot_run_sum |
  | -------------- | -----------: | -----------: | ----------: | ----------: |
  | macbook m1 max |         0.48 |        20.68 |        0.43 |       18.60 |
  | macbook neo    |         1.39 |        59.73 |        1.26 |       54.27 |
  | c8gd.4xlarge   |         0.51 |        22.04 |        0.24 |       10.36 |
  | c5ad.4xlarge   |         1.29 |        54.14 |        0.55 |       22.91 |
  | c6a.4xlarge    |         3.37 |       145.08 |        1.11 |       47.86 |
  | c8g.metal-48xl |         3.95 |       169.67 |        0.10 |        4.35 |

I know this is moving the goalpost, however, it's quite interesting that both of these cloud instances with instance-attached storage are still outperformed by the M1 Max (which is 4+ years old) on the cold run. And they would quite likely lose against the latest MacBook Pro with the M5 Pro/Max on both the cold and the hot runs. But that's an experiment for another day.

I ran TPC-DS SF300 now on the c6a.4xlarge. It turns out that it's still quite limited by the EBS disk's IO: while 32 GB memory is much more than 8 GB, DuckDB needs to spill to disk a lot and this shows on the runtimes. Running all 99 queries took 37 minutes, so about half of the MacBook's 79 minutes.

> Command being timed: "duckdb tpcds-sf300.db -f bench.sql"

> Percent of CPU this job got: 250%

> Elapsed (wall clock) time (h:mm:ss or m:ss): 37:00.96

> Maximum resident set size (kbytes): 25559652

Comments URL: https://news.ycombinator.com/item?id=46204891

Points: 2

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=45291313

Points: 2

# Comments: 0

(I work at DuckDB Labs.)

> About the COPY statement, it means we can drop Parquet files ourselves in the blob storage ?

Dropping the Parquet files on the blob storage will not work – you have to COPY them through DuckLake so that the catalog databases is updated with the required catalog and metadata information.

(diclaimer: I work at DuckDB Labs)

https://blobs.duckdb.org/slides/goto-amsterdam-2024-duckdb-g...

It works as a replacement / complementary component to dataframe libraries due to it's speed and (vertical) scalability. It's lightweight and dependency-free, so it also works as part of data processing pipelines.

> The repository does not contain the source code for the frontend, which is currently not available as open-source. Releasing it as open-source is under consideration.

Yes, if you break the file into parts with GNU Parallel, you can easily beat DuckDB as I show in the blog post.

That said, I maintain that it's surprising that DuckDB outperforms wc (and grep) on many common setups, e.g., on a MacBook. This is not something many databases can do, and the ones which can usually don't run on a laptop.

On the setup side, I agree that local (instance-attached) disks should be preferred but does EBS incur an IO fee? It incurs a significant latency for sure but it doesn't have a per-operation pricing:

> I/O is included in the price of the volumes, so you pay only for each GB of storage you provision.

(https://aws.amazon.com/ebs/pricing/)

Comments URL: https://news.ycombinator.com/item?id=41741410

Points: 2

# Comments: 0