TOAST columns: Artie has automatic detection built in. If a TOAST column hasn't changed, its value won't appear in the WAL. Artie detects this and skips the update for that column in the destination. This works without needing to set REPLICA IDENTITY FULL on your tables.

Schema drift: Artie never requires a schema registry. For relational sources like Postgres, Artie reads the source schema directly and syncs new columns immediately. For DDL changes, Artie uses lazy schema evaluation. On the next DML event for the table, it compares source vs. destination schema and applies any outstanding changes before writing the row.

Let me know if you have any other questions!

The last time I posted here, people had to book a call with us in order to access Artie. Today, that’s no longer the case. You can now connect your source and destination and start streaming immediately.

I spent years of my career building large-scale data pipelines and experienced how difficult it was to get real-time data firsthand. I believed there must be a better way to stream data into our warehouse, which resulted in Artie being born. And now with AI agents, reducing data latency has become more and more crucial as agents need to make decisions off of fresh data.

When I first started building Artie, I quickly learned that the components meant to keep CDC running smoothly are very much bolted on with tons of edge cases. Unfortunately in practice, they were not built to work together. We ended up dealing with schema drift, backfill race conditions, Kafka offset commits, and TOAST columns. I’d love to know if others have hit these same issues while building in-house.

artie.com, would love feedback!

Comments URL: https://news.ycombinator.com/item?id=48471805

Points: 21

# Comments: 5

I encountered this problem firsthand as a heavy data warehouse user at prior jobs. Our warehouse data was always lagged and analytics were always stale. The most visceral version of this today: imagine an AI agent making decisions – on pricing, support routing, risk scoring – off a data warehouse that's 3-12 hours behind.

When we started, I thought the hard part was reading the WAL. The real problems:

Schema drift: CDC events carry row data but not column metadata, so when an engineer adds a column in prod, events with that column start arriving at the destination before you've run ALTER TABLE. In this case, you wouldn’t get an error – you would just silently drop data.

Backfill race conditions: the typical approach (snapshot first, then start CDC) means by the time your snapshot finishes on a large table, the stream has moved on. If you stitch them together wrong, you overwrite newer data with older snapshots.

Kafka offset commits: this sounds obvious but they’re difficult to execute. You can only commit after a successful merge into the destination, or you double-write on replay. Partial failures across a distributed system compound this quickly.

TOAST columns: Postgres omits unchanged TOAST columns (large text/JSON/bytea – think JSONB config fields, long descriptions, binary blobs) from WAL events entirely for storage optimization. A naive pipeline reads ‘missing’ as ‘set to null’ and silently wipes valid data, which can mean a customer's entire config blob gets wiped out because an unrelated column on the same row got updated. The fix is merge logic that treats absent columns as ‘don't touch’ rather than ‘set to null,’ which breaks most off-the-shelf UPSERT patterns.

Curious whether others have hit these same walls building in-house, and would love feedback.

Comments URL: https://news.ycombinator.com/item?id=48464686

Points: 3

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=38280664

Points: 32

# Comments: 7

Comments URL: https://news.ycombinator.com/item?id=37468327

Points: 4

# Comments: 1

We found it far more efficient and less risky to do CDC streaming and snapshotting w/o read lock in parallel to two different topics. Once snapshot is done and drained, we then move to drain the CDC topic.

GitHub Issue: https://github.com/artie-labs/transfer/issues/157

If you'd like to be another design partner for us on this, do reach out. I'm at robin@artie.so.

We will be creating a S3 destination with TSV, Avro, Parquet format support verily soon.

For MySQL and MongoDB, we rely on Debezium to perform the initial snapshots.

Your comment regarding DDL is interesting.

Today, this is what happens:

1/ Column doesn't exist in the destination, we'll create it based on our typing inference from the data type (important: not the data value).

2/ Certain tools will handle automatic column data type conversion if a change like this was detected at the source. We do not do this. We will simply hard fail and cause head-of-line blocking reasons being: this is anti-pattern and should be rare, in which case - it's okay to cause an err and require manual intervention for this breaking change.

3/ If the column has been dropped from the source, you as the end user can decide whether this column should be also dropped in the destination, or not. The default is not to drop it.

^ We hear more customers explicitly don't want columns to be dropped because it could cause downstream errors, such as other views / tables not compiling due to referencing a non-existent column.

We haven't heard much from folks that don't even want columns to be added. If there is a need, we can definitely add that as a config option to provide maximum configurability.

> Finally, the biggest issue with CDC always ends up being the seed loads, recoveries and the incremental snapshot strategies.

Yep totally. On the recovery bit, this is exactly why we are leveraging Kafka. If there are any particular issues, we simply don't commit the offset and cause head-of-line blocking.

On the incremental snapshot and recoveries bit, we primarily leverage Debezium's DDD-3 high watermark strategy [1] for MySQL and MongoDB. Postgres has a different issue in that replication slots can grow really fast, esp on AWS! [2]. We ended up writing our own custom snapshotter for Postgres that is Debezium compatible to onboard customers that have a massive dataset and cannot afford to have a read lock on their WAL.

[1] https://github.com/debezium/debezium-design-documents/blob/m... [2] https://www.morling.dev/blog/insatiable-postgres-replication...

What I am proposing above is ways to provide a view to teams that do not want real-time data while keeping your underlying dataset in real-time.

> Something McKinley doesn't address is that it's quite advantageous if the values in your data warehouse don't change intra-day because this lets business users reach consensus. Whereas if Bob runs a report and gets $X, and Alice runs the same report 5 minutes later and gets $Y, that creates confusion (much more than you would expect). I recall a particular system I built that refreshed every 6 hours (limited by upstream), that eventually Marketing asked me to dial back to every 24 hours because they couldn't stand things changing in the middle of the day.

If they want to see a consistent view of the report, you could bound this.

1/ SELECT * FROM FOO WHERE DATE_TRUNC('day', updated_at) < DATE_TRUNC('day', DATEADD(day, -1, CURRENT_DATE()));

If your dataset doesn't contain kv, you can turn on include `artie_updated_at` which will provide an additional column with the updated_at field to support incremental ingestion.

2/ If you had stateful data, you could also explore creating a Snowflake task and leveraging the time travel f(x) to create a "snapshot" if your workload depended on it.

3/ Also, if you _did_ want this to be more lagged, you can actually increase the flushIntervalSeconds [1] to 6h, 24h, whichever time interval you fancy. You as the customer should have maximum flexibility when it comes to when to flush to DWH.

4/ You can also choose to refresh the analytical report on Looker / Mode to be daily. [2]

> Now of course I see you're targeting more real-time use cases like fraud detection. That's great! But why you would run a fraud detection process out of your data warehouse, which likely doesn't even have a production-grade uptime SLA? Run it out of your production database, that's what it's for!

You can certainly do this in production db (that was our original hypothesis as well!), however, after talking to more companies...it has become more obvious to us that folks that are running fraud algos actually want to join this across various data sets. Further, by using a DWH - it provides a nice visualization layer on top.

Of course, you could go with something even more bespoke by utilizing real-time DBs such as Materialize / Rockset / RisingWave. Just comes with trade offs such as increase in architectural complexity.

There are also plenty of additional use cases this can unlock given that DWH is a platform, any post-DWH application can benefit from less lag, such as reverse ETLs.

[1] https://docs.artie.so/running-transfer/options

[2] https://mode.com/help/articles/report-scheduling-and-sharing...

I ask because there's a workaround by setting `binary.handling.mode` to a STRING type [1].

Transfer will then automatically pick this up and write this as a B64 string to the DWH.

[1] https://debezium.io/documentation/reference/stable/connector...