For example, with Multigres, you should be able to achieve true zero downtime major version upgrade by simply resharding [2]. With vanilla Postgres + pgBouncer, you can only achieve near-zero downtime (few seconds at most), though it's probably good enough for most use cases.

[0] https://multigres.com/

[1] https://fpaxos.github.io/

[2] https://multigres.com/docs#migrate-across-postgres-versions

[1] https://github.com/polynya-dev/pg2iceberg

[2] https://www.polarsignals.com/blog/posts/2024/05/28/mostly-ds...

I'm guessing with Neon, since their storage is a lakehouse, you get this for free.

[0] https://www.orioledb.com/

[0] https://github.com/deepnote/deepnote

[0] https://vega.github.io/vega-lite/examples

[0] https://vega.github.io/vega-lite/

- I'm writing this service even though team X has built the same thing, because my team lead doesn't trust team X since the last time we depended on their service 3 years ago, they had a major downtime that screwed us up big time

- This service is using AWS Lambda simply because I think it's cool, despite the fact that the company has a dedicated team running k8s stack with argocd, argo rollouts, KEDA, etc for the entire company

- Service Y is written in this particular way because it's a service that is shared with another team that came from a company that was acquired, and they wouldn't use it unless we write it this particular way, and making the top execs happy is more important than dealing with a small tech debt (this is probably true)

But no one is going to write these in their RFC. But Sarah knows.

[0] https://multigres.com/ [1] https://neki.dev/

The solution: Polynya replicates your data into Iceberg, and gives your agent an ephemeral ClickHouse instance on demand. Polynya also provides persistent workspaces — collections of views that survive across sessions. So from your agent's point of view, it's a 24/7 data warehouse.

At its core, Polynya is a data platform for streaming real-time data from Postgres to Iceberg. But instead of spending hours setting up costly and complex pipelines involving Kafka, Debezium, Flink, etc., Polynya lets you do this with just one command:

npx polynya create

It's currently free on early access, so do try it out and give me any feedback! Currently there is no web dashboard yet (coming soon), you interact with it 100% through CLI.

P.S. Polynya is built on top of pg2iceberg, the open source Postgres to Iceberg replication tool I've been building for the past few weeks: https://pg2iceberg.dev/

Comments URL: https://news.ycombinator.com/item?id=47914821

Points: 5

# Comments: 0

One example was an engineer wanted to build an API that accepts large CSV (GBs of credit reports) to extract some data and perform some aggregations. He was in the process of discussing with SREs on the best way to process the huge CSV file without using k8s stateful set, and the solution he was about to build was basically writing to S3 and having a worker asynchronously load and process the CSV in chunks, then finally writing the aggregation to db.

I stepped in and told him he was about to build a data warehouse. :P

- Read access through OLAP, not OLTP. You just need to setup a near real-time replication between your OLTP and OLAP.

- Write access through API, just like your application. You can add fancy things like approval layer, e.g. you agent cannot "ban_user(id)", but it can "request_to_ban_user(id)", and the action only happens once you approve it.

Based on my experience with Claude, it's pretty damn good at doing data analysis, if given the right curated data models. You still need to eyeball the generated SQL to make sure it makes sense.

> and how?

1. Replicate your Postgres into Snowflake/Databricks/ClickHouse/etc, or directly to Iceberg and hook it up to Snowflake/Databricks/ClickHouse/etc.

2. Give your agent read access to query it.

3. Build dimensional models (facts and dimensions tables) from the raw data. You can ask LLM for help here, Claude is pretty good at designing data models in my experience.

4. Start asking your agent questions about your data.

Keep steps 3-4 as a tight feedback loop. Every time your agent hallucinates or struggle to answer your questions, improve the model.

Side note: I'm currently building a platform that does all 3 (though you still need to do 2 yourself), you just need Postgres + 1 command to set it up: https://polynya.dev/