The only difference is that a random asshole on the street can't really do anything to me, I can just walk away.

A random asshole cop has a crazy amount of discretionary power over me, even without him stepping outside departmental procedures or the law.

Very odd. I use passkeys extensively with BitWarden and I love it to the point where it's my preferred way of securing things at this point.

The fact that the website presents the question to BitWarden in a structured way (what website, what username) means that I never fight with selecting the right account to get the password for (because I commonly have multiple accounts for a single site) and it generally makes the login flow much smoother.

Environment is MacOS with Brave/Firefox + iOS.

Unless you're required to retain logs for some reason like a litigation hold or legally or contractually mandated retention period and you violate those, while the adversarial party might be annoyed at you for not retaining logs there isn't much they can or will do beyond being annoyed.

Of course if you destroy logs after being notified of litigation or inquiry, you're gonna have a bad day.

When voltage is dropping (or phase is falling out of alignment) don't you want to shed load to stabilize? The issue seems to be more that too much load could shed too quickly causing wild oscillations in grid conditions from an undervolt to an overvolt. Seems like the correct, and not terribly complicated, solution to this is to have a process whereby the grid operator can request large customers to shed load and move to backup power temporarily.

That seems like such a reasonable suggestion that I would be shocked if such a thing does not already exist and is simply just not reactive enough and too manual.

I'm very happy for SpaceX's success but a monopoly on space launch capacity benefits nobody, SpaceX included.

SpaceX was launching a modest % of the LEO constellation but after the Blue Origin failure, SpaceX is the only launch provider who can fill that gap and actually let LEO deliver on contracted time.

Please don't misunderstand me, I'm no Musk sycophant though I do love SpaceX and Starlink. I want us to have multiple providers of super cheap space launch capabilities and multiple diverse LEO satellite constellations (3-4 on a global scale makes sense I think?).

I'm sure BlueOrigin will get there some day and I'm sure LEO will get there too (maybe even in the 2028 window if they expand their SpaceX launch partnership).

In fact the article comes dangerously close to admitting that there is correlation without correlation, it opens with:

> Here is the short version. In 2012, Caltrain budgeted its electrification project — the backbone of the Peninsula's transit future and a prerequisite for high-speed rail to ever reach San Francisco — at roughly $1.5 billion. By 2017 that number had ballooned to $1.9 billion. In between, the Town of Atherton sued.

While I don't agree with what Atherton did here (in general, I did not look at the specifics), you have to be fairly negligent to think you're going to build something in California without a massive legal headache. This is a legislative problem which it sounds like, for this narrow case, the legislature actually solved (shockingly to me). I find it hard to blame the residents of the city for exercising their rights.

Right, exactly, I highly doubt the facility went into any kind of actual uncontrolled thermal rise. This is news because they had to take such drastic actions. I'm sure its common that they force spot prices up (probably way up) to compensate for reduced capacity due to events, I'm sure they even sometimes fake no capacity for similar reasons. No capacity means "I don't want to turn on your node" not merely "I don't have any more physical servers I could turn up for you".

This is news because they powered off some non-preemptible customer loads, which actually makes me wonder if you saw that chain of events occur here.

spot prices rise -> new instance availability goes to 0 -> preemptible instances go dark -> normal instances go dark.

Its cold up here in the winter, sadly, the residual heat from even totally passive components like switch gear is enough to warm things up enough to attract them. .001% of 1MW of power is still quite warm. (I have no idea how much switchgear leaks but i know they are warm even in winter outdoors).

And, yeah, the rest of the writeup is also an amalgamation of some panic-inducing experiences in my life.

Some fail below 100% too.

A decade ago it was trivial to just tell the hypervisor to reduce the cpu fraction of all VMs by half and leave half unallocated. Now, it's much more complicated and definitely would be user visible.

But this is the physical world, shit happens.

The algorithm didn't know that fuse was lose and fine at 50% duty cycle but was high resistance and going to blow at 100%.

Cooling in datacenters is like everything else both over and under provisioned.

It's overprovisioned in the sense that the big heat exchange units are N+1 (or in very critical and smaller load facilities 2N/3N). This is done because you need to regularly take these down for maintenance work and they have a relatively high failure rate compared to traditional DC components and require mechanical repairs that require specialized labor and long lead times. In a bigger facility its not uncommon to have cooling be N+3 or more when N becomes a bigger number because you're effectively always servicing something or have something down waiting for a blower assembly which needs to be literally made by a machinist with a lathe because that part doesn't exist anymore but that's still cheaper than replacing the whole unit.

The system are also under-provisioned in the sense that if every compute capacity in the facility suddenly went from average power draw to 100% power draw you would overload the cooling capacity, you would also commonly overload things in the electrical and other paths too. Over provisioning is just the nature of the industry.

In general neither of these things poses a real problem because compute loads don't spike to 100% of capacity and when they do spike they don't spike for terribly long and nobody builds facilities on a knife-edge of cooling or power capacity.

The problem comes when you have the intersection of multiple events.

You designed your cooling system to handle 200% of average load which is great because you have lots of headroom for maintenance/outages.

Repair guy comes on Tuesday to do work on a unit and finds a bad bearing, has to get it from the next state over so he leaves the unit off overnight to not risk damaging the whole fan assembly (which would take weeks to fabricate).

The two adjacent cooling units are now working JUST A BIT harder to compensate and one of them also had a motor which was just slightly imbalanced or a fuse which was loose and warming up a bit and now with an increased duty cycle that thing which worked fine for years goes pop.

Now you're minus two units in an N+2 facility. Not really terrible, remember you designed for 200% of average load.

That 3rd unit on the other side of the first failed unit, now under way more load, also has a fault. You're now minus 3 in a N+2 facility.

Still, not catastrophic because really you designed for 200% of average load.

The thing is, it's now 4AM, the onsite ops guy can't fix these faults and needs to call the vendor who doesn't wake up till 7AM and won't be onsite till 9.

Your load starts ramping up.

Everything up above happens daily in some datacenter in the USA. It happens in every datacenter probably once a year.

What happens next is the confluence of events which puts you in the news.

One of your bigger customers decides now is a great time to start a huge batch processing job. Some fintech wants to run a huge model before market open or some oil firm wants to do some quick analysis of a new field.

They spin up 10000 new VMs.

Normally, this is fine, you have the spare capacity.

But, remember, you planned for 200% of AVERAGE cooling capacity and this is not nodes which are busy but not terribly busy, these are nodes doing intense optimized number crunching work which means they draw max power and thus expel max waste heat.

Not only has your load in terms of aggregate number of machines spiked but their waste heat impact is also greater on average.

Boom, cascading failure, your cooling is now N-4.

Server fans start ramping up faster which consumes more power.

Your cooling is now N-5.

Alarms are blaring all over the place.

Safeties on the cooling units start to trip as they exceed their load and refrigerant pressures rise.

Your cooling is now N-6.

Your cooling is now N-7.

Your cooling is now 0.

City: just take a walk through manhattan and in a block or two look at the giant open-pit excavation with a 200-year-old morass of undocumented infrastructure under the street. This is before you even try to run fiber up to units in buildings which were built before electricity was standard. I am hardly saying it can't be done, simply that it is not as easy as density makes it seem.

State: the exact opposite problem -- just drive two hours north of NYC and (if you're not still in manhattan) you'll be in some fantastic areas of the state, but, the exact opposite problem exists.

Of note, I do think both of these problems are solvable and we should fundamentally solve them. Just anybody who thinks it's easy or cheap to do so is being myopic. If spent wisely, could be a very useful investment of our money, however.