Using WAL → CDC → downstream systems keeps the architecture simple at first, and tools like Debezium make it relatively straightforward to pipe those changes into Kafka or other processors.

Where things start getting interesting is the analytics side. Once the CDC stream lands in something like Iceberg tables, you effectively get a continuously updated analytical dataset that can be queried with engines like Spark or Trino.

At that point the architecture starts to look less like a traditional “data warehouse pipeline” and more like a streaming-first lakehouse where operational data flows directly into analytical storage.

The main challenge I’ve seen is operational complexity once you start combining: CDC ingestion stream processing lakehouse storage (Iceberg/Delta) distributed query engines That’s where platforms trying to package the open stack together (e.g. Spark + Iceberg + Trino) become interesting. Some newer platforms like IOMETE are basically trying to simplify running that type of lakehouse stack on Kubernetes so teams don’t have to glue everything together manually.

Curious where people think the breakpoint is at what scale does Postgres+CDC stop being “good enough” and you start needing a dedicated log system as the primary event backbone?

LLMs can generate queries, transformations, or even Spark jobs that look reasonable but if the underlying data contracts, schema expectations, or evaluation criteria aren’t defined, you end up with something that looks correct but is semantically wrong.

In practice, the teams that get the most value from AI-assisted development tend to have: clearly defined datasets reproducible data pipelines well-defined outputs / metrics Once those pieces are in place, AI becomes much more useful because it’s operating inside a structured system instead of guessing context. That’s also why there’s been a lot of interest lately in lakehouse-style platforms that combine data engineering, analytics, and AI workflows in one place (e.g. platforms like IOMETE).

When the data layer is structured and reproducible, AI tooling becomes far more reliable. Curious if others here have seen the same pattern when using LLMs for data engineering or analytics work.

Curious how you handle schema drift at the introspection phase specifically when an API starts returning a field as sometimes a string and sometimes an object depending on the endpoint response. Does Forge pick a winner or surface it as a conflict for the user to resolve?

(Disclosure: I work at IOMETE where we think about this a lot. I am happy to go deeper if useful.)