Comments URL: https://news.ycombinator.com/item?id=47867970

Points: 4

# Comments: 0

> This is why free -h on a Linux box can look alarming. You see almost no “free” memory, but most of it is “available” - and the page cache is using it.

And other buffers and stuff too. This is a great thing on bare metal, because it's a bet that the marginal cost of using an empty memory page is zero. This is true on bare metal, always. But in containers, or multi-tenant infrastructure, that isn't true anymore. That's where stuff like DAMON come in: https://www.kernel.org/doc/html/v5.17/vm/damon/index.html

In Aurora Serverless this kind of page cache management is a critical part of what the control plane does. Essentially we need to size the page cache to be big enough for great performance, but small enough not to cost the customer unnecessarily. We go into quite a lot of detail on that in our VLDB'25 paper: https://assets.amazon.science/ee/a4/41ff11374f2f865e5e24de11...

> Linux fills free memory with page cache on purpose. It’s a bet: if someone reads this block again, I already have it.

This works because most database workloads have great temporal and spatial locality. And it works well. But it's also one of the biggest practical issues people run into with relational databases in production: performance is great until it isn't. The shared buffers and page cache keep reads to near zero, but when the working set grows even a tiny bit bigger, then the rate of reads can go up super quickly.

This is why in both Aurora Serverless and Aurora DSQL we do buffer and cache sizing very dynamically, getting rid of this cliff for most workloads.

Inspired me to write this blog post: https://brooker.co.za/blog/2023/04/20/hobbies.html

Unless you're tapping something super tough (306?), Amazon taps are fine for hand tapping. Go in straight, use a good lubricant.

- Aurora storage scales with your needs, meaning that you don't need to worry about running out of space as your data grows. - Aurora will auto-scale CPU and memory based on the needs of your application, within the bounds you set. It does this without any downtime, or even dropping connections. You don't have to worry about choosing the right CPU and memory up-front, and for most applications you can simply adjust your limits as you go. This is great for applications that are growing over time, or for applications with daily or weekly cycles of usage.

The other Aurora option is Aurora DSQL. The advantages of picking DSQL are:

- A generous free tier to get you going with development. - Scale-to-zero and scale-up, on storage, CPU, and memory. If you aren't sending any traffic to your database it costs you nothing (except storage), and you can scale up to millions of transactions per second with no changes. - No infrastructure to configure or manage, no updates, no thinking about replicas, etc. You don't have to understand CPU or memory ratios, think about software versions, think about primaries and secondaries, or any of that stuff. High availability, scaling of reads and writes, patching, etc is all built-in.

The feedback loop is replicas slow -> traffic increases to primary -> primary slows -> replicas slow, etc. The only way out of this loop is to shed traffic.

It helps in the case where clients are (a) able to contact a minority partition, and (b) can tolerate eventual consistency, and (c) can’t contact the majority partition. These cases are quite rare in modern internet-connected applications.

Consider a 3AZ cloud deployment with remote clients on the internet, and one AZ partitioned off. Most often, clients from the outside will either be able to contact the remaining majority (the two healthy AZs), or will be able to contact nobody. Rarely, clients from the outside will have a path into the minority partition but not the majority partition, but I don’t think I’ve seen that happen in nearly two decades of watching systems like this.

What about internal clients in the partitioned off DC? Yes, the trade-off is that they won’t be able to make isolated progress. If they’re web servers or whatever, that’s moot because they’re partitioned off and there’s no work to do. Same if they’re a training cluster, or other highly connected workloads. There are workloads that can tolerate a ton of asynchrony where being able to continue while disconnected is interesting, but they’re the exception rather than the rule.

Weak consistency is much more interesting as a mechanism for reducing latency (as DynamoDB does, for example) or increasing scalability (as the typical RDBMS ‘read replicas’ pattern does).

Note that I’m not saying that partitions don’t happen. They do! But in typical internet connected applications the cases where a significant proportion of clients is partitioned into the same partition as a minority of the database (i.e. the cases where AP actually improves availability) are very rare in practice.

For client devices and IoT, partitions off from the main internet are rare, and there local copies of data are a necessity.

Event sourcing architectures can be great, but they also tend to be fairly complex (a lot of moving parts). The bigger practical problem is that they make it quite hard to offer clients ‘outside the architecture’ meaningful read-time guarantees stronger than a consistent prefix. That makes clients’ lives hard for the reasons I argue in the blog post.

I really like event-based architectures for things like observability, metering, reporting, and so on where clients can be very tolerant to seeing bounded stale data. For control planes, website backends, etc, I think strongly consistent DB architectures tend to be both simpler and offer a better customer experience.

* Ok, there’s one edge case in the cross-shard commit protocol where two committers can deadlock, which needs to be resolved by aborting one of them (the moral equivalent of WAIT-DIE). This never happens with single-shard transactions, and can’t be triggered by any SQL patterns.

The point of that section, which maybe isn’t obvious enough, is to reflect on how eventually-consistent read replicas limit the options of the database system builder (rather than the application builder). If I’m building the transaction layer of a database, I want to have a bunch of options for where to send me reads, so I don’t have the send the whole read part of every RMW workloads to the single leader.

By pushing things like read-scale-out into the core database, and away from replicas and caches, we get to have stronger client and application guarantees with less architectural complexity. A great combination.

But I agree with you in general - the dynamic nature of systems means, in my mind, that you need to use client-side guarantees, rather than state guarantees, to reason about this stuff in general. State guarantees are nicer to prove and work with formally (see Adya, for example) while client side guarantees are trickier and feel less fulfilling formally (see Crooks et al “Seeing is Believing”, or Herlihy and Wing).

Linearizability, as a global guarantee, is much nicer because it avoids all those edge cases.

The Gilbert and Lynch definition of CAP calls this state ‘unavailable’, in that it’s not available to all clients. Practically, though, it’s still available for two thirds of clients (or more, if we can reroute clients from the outside), which seems meaningfully ‘available’ to me!

If you don’t believe me, check out Phil Bernstein’s paper (Bernstein and Das) about this. Or read the Gilbert and Lynch proof carefully.

Specifically, these definitions require that transactions appear to execute in some serial order, and place no constraints on that serial order. So the database can issue all reads at time zero, returning empty results, and all writes at the time they happen (because who the hell cares?).

The lesson? Demand real-time guarantees.

I notice a similar 'holographic' effect when coloring titanium a couple weeks back, and experimented with getting them dialed in along the same lines as this video. I didn't have nearly as much success, despite the underlying physics being similar. My guess is that the much lower thermal conductivity of titanium causes a lot more smudging than on stainless, which makes the grating effect less pronounced.

One interesting thing I noted with Ti is that satin finished Ti (media blasted with 500 grit glass media) won't take a color from electrocoloring, but will from MOPA laser coloring. Not nearly as nice as polished Ti, but still there. Given that they are such similar processes (growing a set thickness oxide layer), its somewhat surprising to see different results.

I guess I'm going to have to experiment on some polished 304.