Hacker News: AdamProut

New comment by AdamProut in "Ok, but does it scale?"

AdamProut — Fri, 04 Sep 2026 15:07:31 +0000

I'll admit to not reading your code. It's hard to keep up with all the different databases launched in the last decade.

I have doubts that a global readwrite lock around a hashtable makes for a good general purpose storage system.

New comment by AdamProut in "Ok, but Does It Scale?"

AdamProut — Fri, 04 Sep 2026 15:00:34 +0000

what kind of maintenance?

I think one of the maybe less talked about benefits of distributed SQL is support for nearly transparent rolling upgrades of the database with very little impact to a running workload. Spanner is best in class at this.

New comment by AdamProut in "Ok, but does it scale?"

AdamProut — Fri, 04 Sep 2026 14:26:31 +0000

The intro section is a good summary of why distributed SQL databases (Spanner, roach, Yugabyte, TiDB) haven't taken off in the market in the same way as say distributed data warehouses have (Snowflake, Databricks, FabricDW, Clickhouse, etc.).

I would add a few other things to the list of scaling problems. Some SQL features are hard to scale out (auto_increment/serial columns, unique secondary keys, foreign keys, etc.). Some SQL query operators are hard to scale out for OLTP queries that want low latency and high throughput (DISTINCT, LIMIT/TOP-N, non-collocated joins). I take it SpacetimeDB is a nosql database, so these problems are less important to them?

As to how spacetimedb plans to scale out, I didn't follow it fully. It's hard to take the spacetimedb folks seriously (see: https://strn.cat/posts/spacetime/).

New comment by AdamProut in "Agents Are Better Testers Than We Are"

AdamProut — Thu, 30 Apr 2026 23:05:57 +0000

Agents should be driving up the quality of software if we focus them on finding and fixing bugs.

Agents Are Better Testers Than We Are

AdamProut — Thu, 30 Apr 2026 23:05:57 +0000

Article URL: https://medium.com/@adamprout/agents-are-better-testers-than-we-are-30b1738114d6

Comments URL: https://news.ycombinator.com/item?id=47969436

Points: 1

# Comments: 1

New comment by AdamProut in "Show HN: Pg_mooncake – Delta/Iceberg columnstore tables in Postgres"

AdamProut — Wed, 30 Oct 2024 20:15:49 +0000

Very cool! pg_duckdb itself is missing fully integrated storage - it can query data stored externally (say in S3) in delta/iceberg formats, but it can't write out data in those formats via transactional writes to PG tables (insert\update\deletes). pg_mooncacke is one neat way of solving that problem. It lets you have a columnstore table in Postgres that can do both reads and writes as if it's any other PG table and have the storage format be an open format like delta/iceberg/etc with that data persisted to blob store (like most cloud DWs would do anyways).

New comment by AdamProut in "Pg_parquet: An extension to connect Postgres and parquet"

AdamProut — Thu, 17 Oct 2024 21:33:05 +0000

Had a similar thought. Azure Postgres has something similar to pg_parquet (pg_azure_storage), but we're looking into replacing it with pg_duckdb assuming the extension continues to mature.

It would be great if the Postgres community could get behind one good opensource extension for the various columnstore data use cases (querying data stored in an open columnstore format - delta, iceberg, etc. being one of them). pg_duckdb seems to have the best chance at being the goto extension for this.

New comment by AdamProut in "A write-ahead log is not a universal part of durability"

AdamProut — Mon, 01 Jul 2024 14:19:51 +0000

Checksums can detect a torn page, but not always repair them. It's likely a good part of the database page is gone (i.e., an amount of data that matches the disk / file system atomic write unit size is probably missing). Torn page writes are a pretty common scenario too, so databases need to be able to fully recover from them - not just detect them and report a corruption (ie., just pull the power plug from the machine during a heavy write workload and you're likely to get one - it doesn't require a solar ray to flip a bit :) ).

New comment by AdamProut in "A write-ahead log is not a universal part of durability"

AdamProut — Mon, 01 Jul 2024 13:27:00 +0000

Maybe good to mention torn pages somewhere too? Both MySQL and Postgres jump through some hoops to both detect them and repair them [1][2]. So, even the scenario in the post where fsync is used to harden writes, the database still needs to handle torn pages (or requires using a file system \ storage that guarantees atomic page writes at the page size the database is using as several managed\cloud databases do).

[1] https://wiki.postgresql.org/wiki/Full_page_writes [2] https://dev.mysql.com/doc/refman/8.0/en/innodb-doublewrite-b...

New comment by AdamProut in "Headline Driven Development"

AdamProut — Mon, 01 Apr 2024 15:14:16 +0000

I'm not sure why you're getting downvoted. I think the pressure to appease Gartner usually starts when companies bring in CEOs whose primary background is enterprise sales. They tend to over value magic quadrant positioning (in my view).

New comment by AdamProut in "Does MongoDB Support ACID Transactions?"

AdamProut — Tue, 16 Jan 2024 21:37:03 +0000

Based on the writings of 2 Turing awards winners for work on databases; No MongoDB doesn't have ACID transactions.

Does MongoDB Support ACID Transactions?

AdamProut — Tue, 16 Jan 2024 21:37:03 +0000

Article URL: https://medium.com/@adamprout/which-ansi-sql-isolation-level-is-required-to-be-acid-compliant-or-does-mongodb-actually-support-5518a3b06258

Comments URL: https://news.ycombinator.com/item?id=39019498

Points: 1

# Comments: 2

New comment by AdamProut in "PID Controllers in Unity3D (2021)"

AdamProut — Tue, 16 Jan 2024 21:19:07 +0000

Control theory is also used by databases (probably not as often as it should be). It's great for "self tuning" [1], for example tuning the various cache sizes a database has to maximize throughput under changing workload conditions. Its definitely worth spending the time to understand PID controllers if your an engineer working on databases.

[1] https://www.vldb.org/conf/2006/p1081-storm.pdf

New comment by AdamProut in "An overview of distributed Postgres architectures"

AdamProut — Tue, 09 Jan 2024 16:30:08 +0000

RE: "Distributed SQL offers elasticity (no downtime resize"). I'm not sure this is as much of an advantage of distributed databases vs single host databases anymore. Some of the tech to move virtual machines between machines quickly (without dropping TCP connections) is pretty neat. Neon has a blog post about it here[1]. Aurora Serverless V2 does the same thing (but I can't find a detailed technical blog post talking about how it works). Your still limited by "one big host" but its no longer as big of a deal to scale your compute up/down within that limit.

[1] https://neon.tech/blog/scaling-serverless-postgres

New comment by AdamProut in "Blaze: Fast query execution engine for Apache Spark"

AdamProut — Fri, 20 Oct 2023 22:15:27 +0000

Databricks has kept their Photon[1][2] query engine for Spark closed sourced thus far. Unless EMR has made equivalent changes to the Spark runtime they use Databricks should be much faster. Photon brings the standard vectorized execution techniques used in SQL data warehouses for many years to Spark.

[1] https://docs.databricks.com/en/clusters/photon.html [2] https://dl.acm.org/doi/10.1145/3514221.3526054

New comment by AdamProut in "The broad set of computer science problems faced at cloud database companies"

AdamProut — Tue, 22 Aug 2023 03:58:44 +0000

yeah, that is part of the trade off. Using an ephemeral SSD (for a database) means the database needs to have another means of making the data durable (replication, storing data in S3, etc.).

There are AWS instance types (I3en) with large and very fast SSDs (many times higher IOPS then EBS).

New comment by AdamProut in "The broad set of computer science problems faced at cloud database companies"

AdamProut — Tue, 22 Aug 2023 03:52:42 +0000

Its a popular design for SQL Data warehouses. I think almost all of them (snowflake, redshift, etc.) store cold data in S3 and hot data on local disk[1][2].

It works well if the data is stored as immutable files (i.e., A log structure merge tree) or is not index at all (classical columnstores). S3 doesn't provide an efficient way to update a file.

[1] https://dl.acm.org/doi/10.1145/2882903.2903741 (snowflake SIGMOD paper) [2] https://dl.acm.org/doi/10.1145/3514221.3526055 (singlestore SIGMOD paper)

New comment by AdamProut in "Are You Sure You Want to Use MMAP in Your Database Management System? (2022)"

AdamProut — Mon, 03 Jul 2023 03:51:16 +0000

Yeah for workloads with any long running write transactions a single writer design is a pretty big limitation. Say some long running data load (or a big bulk deletion) running along with some faster high throughput key value writes - the big data load would block all the faster key-value writes when it runs.

No "mainstream" database I'm aware of has a global single writer design.

New comment by AdamProut in "Supabase Vector, the Open Source Vector Toolkit for Postgres"

AdamProut — Thu, 08 Jun 2023 14:37:50 +0000

This may depend on if your use case is only vector search in isolation (ANN lookups). In this scenario pgvector is potentially not the best option (https://ann-benchmarks.com/)

That said, using pgvector (or using other SQL databases with vector search support.. many have this capability) will let you do both ANN and your usual SQL filtering and joining (or full text, or.. etc) to produce more hand tuned results to a query. This is something the specialized vector databases don't have much support for.

New comment by AdamProut in "Databases in 2022: A Year in Review"

AdamProut — Tue, 03 Jan 2023 02:26:18 +0000

Yep, SingleStoreDB (formerly MemSQL) is not open source (probably never will be), but it does have many paying customers who have had workloads in production for over a decade at this point.

Also, isn't taking a bet on a very recently launched database as a service based around an open source database also pretty risky? Say for example the very recently launched Clickhouse Inc. service (which your a co-founder of?).