Hacker News: hamilton

New comment by hamilton in "Fast drilldown dashboards from a single Parquet file"

hamilton — Tue, 25 Aug 2026 11:46:01 +0000

Indeed, OLAP cubes keep finding new life. The critical difference here is that there isn't a heavy query engine (just a Parquet reader) and a few http requests to pull from the cube. So it's light on the backend and light on the frontend.

New comment by hamilton in "Fast drilldown dashboards from a single Parquet file"

hamilton — Mon, 24 Aug 2026 16:45:40 +0000

I think it depends on (1) customer expectations for freshness and (2) scale (both for the cubes and for the customer data in toto). There are many types of customer facing dashboards where giving "live data" is a bad idea for them and for you. And recomputation is indeed a problem, but if the volume of data isn't that high to begin with, it's probably easier than setting up an incremental pipeline architecture vs. a grouping set query in DuckDB. But I am not really a data engineer, so perhaps this is naive.

New comment by hamilton in "Fast drilldown dashboards from a single Parquet file"

hamilton — Mon, 24 Aug 2026 15:28:02 +0000

It's a good question. In a sense, the cube is caching, just materialized ahead of time instead of memoized on demand. A result cache still needs a live database behind it for misses; the cube has no misses, since every question the dashboard is designed to answer has data in the cube already. And for this experiment, the goal was to forgo a database to serve the data in the first place.

I provide caveats for when this would work vs. when it doesn't in the post. For a lot of customer-facing dashboards, I think it's probably pretty good.

New comment by hamilton in "Fast drilldown dashboards from a single Parquet file"

hamilton — Mon, 24 Aug 2026 15:00:37 +0000

Agreed, for a public demo like this one, GitHub Pages would work great (or any host that speaks HTTP range requests with CORS). I used R2 partly because I wanted to see how it behaved, and partly because the real use-case doesn't fit Pages. The source data already lives on R2 as Iceberg, the files are per-customer and would probably need auth (signed URLs or a session-checking Worker), and obviously 10k customer cubes on a schedule works better with object PUTs rather than git deploys

New comment by hamilton in "Instant SQL for results as you type in DuckDB UI"

hamilton — Thu, 24 Apr 2025 19:03:31 +0000

We do have plans. It's a question of effort, not business / philosophy.

New comment by hamilton in "Instant SQL for results as you type in DuckDB UI"

hamilton — Thu, 24 Apr 2025 18:13:53 +0000

it can, but it doesn't format. You can even run the ast!

New comment by hamilton in "Instant SQL for results as you type in DuckDB UI"

hamilton — Thu, 24 Apr 2025 17:24:35 +0000

Obviously one advantage of SQL is everyone knows it. But conceptually, I agree. I think [1]Malloy is also doing some really fantastic work in this area.

This is one of the reasons I'm excited about DuckDB's upcoming [2]PEG parser. If they can pull it off, we could have alternative dialects that run on DuckDB.

[1] https://www.malloydata.dev/ [2] https://duckdb.org/2024/11/22/runtime-extensible-parsers.htm...

New comment by hamilton in "Instant SQL for results as you type in DuckDB UI"

hamilton — Thu, 24 Apr 2025 16:46:28 +0000

Definitely something we want too! (I'm the author / lead for the UI)

New comment by hamilton in "Instant SQL for results as you type in DuckDB UI"

hamilton — Thu, 24 Apr 2025 16:31:43 +0000

I agree; one thing that is neat about Instant SQL is for many reasons, you can't do this with in any other DBMS. You really need DuckDB's specific architecture and ergonomics.

New comment by hamilton in "Instant SQL for results as you type in DuckDB UI"

hamilton — Thu, 24 Apr 2025 16:30:07 +0000

Great feedback! Thanks. We agree w/ the red errors. It's not helpful when it feels like your editor is screaming at you.

New comment by hamilton in "Instant SQL for results as you type in DuckDB UI"

hamilton — Thu, 24 Apr 2025 16:28:44 +0000

You should read the post! This is what the feature does.

New comment by hamilton in "Instant SQL for results as you type in DuckDB UI"

hamilton — Thu, 24 Apr 2025 15:30:38 +0000

Can you say more? Where does it error out? Sounds like a bug; if you could post an example query, I bet we can fix that.

New comment by hamilton in "Instant SQL for results as you type in DuckDB UI"

hamilton — Thu, 24 Apr 2025 15:19:11 +0000

What about it is bizarre?

New comment by hamilton in "Instant SQL for results as you type in DuckDB UI"

hamilton — Thu, 24 Apr 2025 15:13:57 +0000

Me too (author of the post here). In fact, I was watching a seasoned data engineer at MotherDuck show me how they would attempt to debug a regex in a CTE. As a longtime SQL user, I felt the pain immediately; haven't we all been there before? Instant SQL followed from that.

New comment by hamilton in "Instant SQL for results as you type in DuckDB UI"

hamilton — Thu, 24 Apr 2025 15:08:54 +0000

Correct. We only enable fast previews for SELECT statements, which is the actual hard problem. This said, at some point we're likely to also add support for previewing a CTAS before you actually run it.

New comment by hamilton in "Instant SQL for results as you type in DuckDB UI"

hamilton — Thu, 24 Apr 2025 15:07:11 +0000

You can use any variation of DuckDB valid syntax that you want! I prefer to put from first just because I think it's better, but Instant SQL works with traditional select __ from __ queries.

New comment by hamilton in "Instant SQL for results as you type in DuckDB UI"

hamilton — Thu, 24 Apr 2025 15:04:32 +0000

Indeed, we are! We worked with DuckDB Labs to add the query_location information, which we're also enriching with the tokenizer to draw a path through the AST to the cursor location. I've been wanting to do this since forever, and now that we have it, there's actually a long tail of inspection / debugging / enrichment features we can add to our SQL editor.

New comment by hamilton in "The DuckDB Local UI"

hamilton — Thu, 13 Mar 2025 12:17:11 +0000

ftr I'm still on X & posting! https://x.com/hamiltonulmer

New comment by hamilton in "The DuckDB Local UI"

hamilton — Wed, 12 Mar 2025 16:54:06 +0000

True; at the moment the UI is not open source. We've talked about releasing the Column Explorer as a standalone component, but haven't been able to prioritize it yet. We'd like to!

New comment by hamilton in "The DuckDB Local UI"

hamilton — Wed, 12 Mar 2025 16:47:18 +0000

(I designed and built the Column Explorer feature)

Observable's column summary feature is very nice! But I do think there's a very common lineage around these kinds of diagnostics which motivated both Observables and ours. See Jeff Heer's profiler paper[1] for more.

I'm very passionate about this area because I think "first mile problems" are underserved by most tools, but they take the longest to work out.

We had to do some gnarly things[2] to make this feature work well; and there's a lot of room to make it scale nicely and cover all DuckDB data types.

[1] http://vis.stanford.edu/papers/profiler [2] https://motherduck.com/blog/introducing-column-explorer/