Comments URL: https://news.ycombinator.com/item?id=47786428

Points: 3

# Comments: 1

DuckLake is great for the lakehouse layer and it's what we use in production. But there's a gap and thats what I'm trying to address with OpenDuck. DuckLake do solve concurrent access at the lakehouse/catalog level and table management.

But the moment you need to fall back to DuckDB's own compute for things DuckLake doesn't support yet, you're back to a single .duckdb file with exclusive locking. One process writes, nobody else reads.

OpenDuck sits at a different layer. It intercepts DuckDB's file I/O and replaces it with a differential storage engine which is append-only layers with snapshot isolation.

I'd actually recommend the simplest option: just write them to Parquet on S3 and query with plain DuckDB. Or you could use Ducklake - https://ducklake.select/

The short version: OpenDuck interposes a differential storage layer between DuckDB and the underlying file. DuckDB still sees a normal file (via FUSE on Linux or an in-process FileSystem on any platform), but underneath, writes go to append-only layers and reads are resolved by overlaying those layers newest-first. Sealing a layer creates an immutable snapshot.

This gives you:

Many concurrent readers: each reader opens a snapshot, which is a frozen, consistent view of the database. They don't touch the writer's active layer at all. No locks contended.

One serialized write path: multiple clients can submit writes, but they're ordered through a single gateway/primary rather than racing on the same file. This is intentional: DuckDB's storage engine was never designed for multi-process byte-level writes, and pretending otherwise leads to corruption. Instead, OpenDuck serializes mutations at a higher level and gives you safe concurrency via snapshots.

So for your specific scenario — one process writing while you want to quickly inspect or query the DB from the CLI — you'd be able to open a read-only snapshot mount (or attach with ?snapshot=) from a second process and query freely. The writer keeps going, new snapshots appear as checkpoints seal, and readers can pick up the latest snapshot whenever they're ready.

It's not unconstrained multi-writer OLTP (that's an explicit non-goal), but it does solve the "I literally cannot even read the database while another process has it open" problem that makes DuckDB painful in practice.

Comments URL: https://news.ycombinator.com/item?id=47761997

Points: 185

# Comments: 36

One of them is obviously AWS Lambda. Two weeks ago we started processing large volume of transactions (We provide wallet and digital assets infrastructure for businesses) and needed to increase our services performance from request time out to memory to even our RDS instance to be able to accommodate the kind of requests we are currently processing.

However yesterday (29th of July) at around 5:04PM we got an email from AWS about unautorized service charges due to some unauthorized activity and could be a potential account hack or compromise.

My heart first sank because I was just waking up from a nap after reviewing the new lambda functions the teams worked on and planning to promote to production and staging server.

The second is this is a very high risk security issues and could be a potential hack. We primarily use MPC base vault to securely store and process transactions. Regardless, this is a very a big issue and had to respond by first of all changing the root passwords, deactivating all API keys, disabling console login access for all other users and removing any type of old keys.

After doing this will at least help secure our account for the main time. But then I was curious as well since the suspected activity is because of the unauthorized service charge, decided to go check all services from each region and end up checking our AWS Bill and carefully looking at the bill.

At the end it's a bill the team expected following our new changes from increasing our lambda function memory to time out and upgrading our RDS instance, the bill basically makes sense. AWS must have mistakenly detected this as unauthorized charge.

However the issue here is it's been 10 hours, and we have literally lost access to AWS Lambda functions the moment they notified us of this issues and one of our core business solutions is down because AWS shut it down.

The customer support experience from the AWS support team is the worst I have seen in my entire life. This is outrageous and never wished anyone to experience this. For over 10 hours, our customers (businesses such as payment gateway and exchanges) can't run some business operations because the lambda service (which cant be easily migrated to another alternative service) is down.

Their team keep saying they are waiting for their internal team response for over hours and no good response yet. They didn't provide a valid reason nor why this is taking too much time to fix. Its like they are lack of empathy and just robots behind the keyboards.

The AWS Support team are very heartless with no sense of urgency and this is not the first time such encounter is happening.

This has taught us a lesson not to build our product to be tightly locked with a cloud provider as migrating would be like starting from scratch and be more complicated.

If you work in AWS, the technical support team or customer support team and you would like to help us get past this. Please you can reach out here: hi[[at]]powr[[dot]]finance.

Thanks.

Comments URL: https://news.ycombinator.com/item?id=31928266

Points: 27

# Comments: 20

Comments URL: https://news.ycombinator.com/item?id=30018351

Points: 1

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=24264104

Points: 1

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=24129736

Points: 1

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=23166198

Points: 2

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=23141162

Points: 1

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=23111955

Points: 1

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=23089709

Points: 2

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=23077993

Points: 1

# Comments: 0