the OP mentioned 1B requests/day, where there are systems handling 1B requests a second.

My take is that you want to move these decisions into the application tier as much as possible as the first line of defense, both because you can make more precise decisions in the application and you can respond much more quickly. You want things like preemption to about a slower moving loop where you are applying much coarser logic to what gets squeezed.

it was definitely a case of making many things worse to do than in a monorepo, but more things viable to do.

right now we are in a whole economy experiment to see how long you can increase margin through price increases and shrinkflation.

Comments are ad-hoc and don't scale, relying on the author to interpret and adhere to the extent of the reviewers approval.

> And if they're not, I would really like to know how the part that I'm reviewing interacts with the rest of the changes.

you are free to look up, down, and around the stack; nobody is hiding the code from you. But in many cases this is just unnecessary.

For the people who work with stacked diffs (in phab/otherwise) this is exactly what they'd consider reviewing a well-curated set of commits one-by-one.

One distinction is that cognitively a unit of review (a PR, a diff) remains a single bound change. Comments are focused on that change and the PR does not grow with size of the feature

Another distinction is the ability to focus each part of the stack to a particular audience. One change may require review from an external team, another may be just your team mate, a third might be the consuming team. By focusing the stack to the different reviewers you can avoid ambiguity about "what a person is signing off on" in the stack.

aside: one thing that would be great for github reviews is the adoption of change ids such that comments persist across reviews with a rebase workflow.

there are obviously plenty of low/sparse call volume services where passive healthchecks would take forever to get signal, or signal is so infrequently collected its meaningless. and even with decent RPS, say 1m RPS distributed between 1000 caller replicas and 1000 callee replicas, that means that any one caller-callee pair is only seeing 1rps. Depending on your noise threshold, a centralized active healthcheck can respond much faster.

There are some ways to improve signal in the latter case using subsetting and aggregating/reporting controllers, but that all comes with added complexity.

For general reliability, you can create partitions of checkers and use quorum across partitions to determine what the health state is for a given dest. This also enables centralized monitoring to detect systemic issues with bad healthcheck configuration changes (i.e. are healthchecks failing because the service is unhealthy or because of a bad healthchecker?)

In industry, I personnaly know AWS has one or two health-check-as-a-service systems that they are using internally for LBs and DNS. Uber runs its own health-check-as-a-service system which it integrates with its managed proxy fleet as well as p2p discovery. IIRC Meta also has a system like this for at least some things? But maybe I'm misremembering.

* for client-side load balancing, it's entirely possible to move active healthchecking into a dedicated service and have its results be vended along with discovery. In fact, more managed server-side load balancers are also moving healthchecking out of band so they can scale the forwarding plane independently of probes.

* for server-side load balancing, it's entirely possible to shard forwarders to avoid SPOFs, typically by creating isolated increments and then using shuffle sharding by caller/callee to minimize overlap between workloads. I think Alibaba's canalmesh whitepaper covers such an approach.

As for scale, I think for almost everybody it's completely overblown to go with a p2p model. I think a reasonable estimate for a centralized proxy fleet is about 1% of infrastructure costs. If you want to save that, you need to have a team that can build/maintain your centralized proxy's capabilities in all the languages/frameworks your company uses, and you likely need to be build the proxy anyways for the long-tail. Whereas you can fund a much smaller team to focus on e2e ownership of your forwarding plane.

Add on top that you need a safe deployment strategy for updating the critical logic in all of these combinations, and continuous deployment to ensure your fixes roll out to the fleet in a timely fashion. This is itself a hard scaling problem.

It also makes it easier to reason about that output as you can avoid awkward iteration in your declarative spec.

at the end of the day with litestream, when you respond back to a client with a successful write you are only guaranteeing a replication factor of 1.

that seems like the tail wagging the dog