When voltage is dropping (or phase is falling out of alignment) don't you want to shed load to stabilize? The issue seems to be more that too much load could shed too quickly causing wild oscillations in grid conditions from an undervolt to an overvolt. Seems like the correct, and not terribly complicated, solution to this is to have a process whereby the grid operator can request large customers to shed load and move to backup power temporarily.

That seems like such a reasonable suggestion that I would be shocked if such a thing does not already exist and is simply just not reactive enough and too manual.

I'm very happy for SpaceX's success but a monopoly on space launch capacity benefits nobody, SpaceX included.

SpaceX was launching a modest % of the LEO constellation but after the Blue Origin failure, SpaceX is the only launch provider who can fill that gap and actually let LEO deliver on contracted time.

Please don't misunderstand me, I'm no Musk sycophant though I do love SpaceX and Starlink. I want us to have multiple providers of super cheap space launch capabilities and multiple diverse LEO satellite constellations (3-4 on a global scale makes sense I think?).

I'm sure BlueOrigin will get there some day and I'm sure LEO will get there too (maybe even in the 2028 window if they expand their SpaceX launch partnership).

In fact the article comes dangerously close to admitting that there is correlation without correlation, it opens with:

> Here is the short version. In 2012, Caltrain budgeted its electrification project — the backbone of the Peninsula's transit future and a prerequisite for high-speed rail to ever reach San Francisco — at roughly $1.5 billion. By 2017 that number had ballooned to $1.9 billion. In between, the Town of Atherton sued.

While I don't agree with what Atherton did here (in general, I did not look at the specifics), you have to be fairly negligent to think you're going to build something in California without a massive legal headache. This is a legislative problem which it sounds like, for this narrow case, the legislature actually solved (shockingly to me). I find it hard to blame the residents of the city for exercising their rights.

Right, exactly, I highly doubt the facility went into any kind of actual uncontrolled thermal rise. This is news because they had to take such drastic actions. I'm sure its common that they force spot prices up (probably way up) to compensate for reduced capacity due to events, I'm sure they even sometimes fake no capacity for similar reasons. No capacity means "I don't want to turn on your node" not merely "I don't have any more physical servers I could turn up for you".

This is news because they powered off some non-preemptible customer loads, which actually makes me wonder if you saw that chain of events occur here.

spot prices rise -> new instance availability goes to 0 -> preemptible instances go dark -> normal instances go dark.

Its cold up here in the winter, sadly, the residual heat from even totally passive components like switch gear is enough to warm things up enough to attract them. .001% of 1MW of power is still quite warm. (I have no idea how much switchgear leaks but i know they are warm even in winter outdoors).

And, yeah, the rest of the writeup is also an amalgamation of some panic-inducing experiences in my life.

Some fail below 100% too.

A decade ago it was trivial to just tell the hypervisor to reduce the cpu fraction of all VMs by half and leave half unallocated. Now, it's much more complicated and definitely would be user visible.

But this is the physical world, shit happens.

The algorithm didn't know that fuse was lose and fine at 50% duty cycle but was high resistance and going to blow at 100%.

Cooling in datacenters is like everything else both over and under provisioned.

It's overprovisioned in the sense that the big heat exchange units are N+1 (or in very critical and smaller load facilities 2N/3N). This is done because you need to regularly take these down for maintenance work and they have a relatively high failure rate compared to traditional DC components and require mechanical repairs that require specialized labor and long lead times. In a bigger facility its not uncommon to have cooling be N+3 or more when N becomes a bigger number because you're effectively always servicing something or have something down waiting for a blower assembly which needs to be literally made by a machinist with a lathe because that part doesn't exist anymore but that's still cheaper than replacing the whole unit.

The system are also under-provisioned in the sense that if every compute capacity in the facility suddenly went from average power draw to 100% power draw you would overload the cooling capacity, you would also commonly overload things in the electrical and other paths too. Over provisioning is just the nature of the industry.

In general neither of these things poses a real problem because compute loads don't spike to 100% of capacity and when they do spike they don't spike for terribly long and nobody builds facilities on a knife-edge of cooling or power capacity.

The problem comes when you have the intersection of multiple events.

You designed your cooling system to handle 200% of average load which is great because you have lots of headroom for maintenance/outages.

Repair guy comes on Tuesday to do work on a unit and finds a bad bearing, has to get it from the next state over so he leaves the unit off overnight to not risk damaging the whole fan assembly (which would take weeks to fabricate).

The two adjacent cooling units are now working JUST A BIT harder to compensate and one of them also had a motor which was just slightly imbalanced or a fuse which was loose and warming up a bit and now with an increased duty cycle that thing which worked fine for years goes pop.

Now you're minus two units in an N+2 facility. Not really terrible, remember you designed for 200% of average load.

That 3rd unit on the other side of the first failed unit, now under way more load, also has a fault. You're now minus 3 in a N+2 facility.

Still, not catastrophic because really you designed for 200% of average load.

The thing is, it's now 4AM, the onsite ops guy can't fix these faults and needs to call the vendor who doesn't wake up till 7AM and won't be onsite till 9.

Your load starts ramping up.

Everything up above happens daily in some datacenter in the USA. It happens in every datacenter probably once a year.

What happens next is the confluence of events which puts you in the news.

One of your bigger customers decides now is a great time to start a huge batch processing job. Some fintech wants to run a huge model before market open or some oil firm wants to do some quick analysis of a new field.

They spin up 10000 new VMs.

Normally, this is fine, you have the spare capacity.

But, remember, you planned for 200% of AVERAGE cooling capacity and this is not nodes which are busy but not terribly busy, these are nodes doing intense optimized number crunching work which means they draw max power and thus expel max waste heat.

Not only has your load in terms of aggregate number of machines spiked but their waste heat impact is also greater on average.

Boom, cascading failure, your cooling is now N-4.

Server fans start ramping up faster which consumes more power.

Your cooling is now N-5.

Alarms are blaring all over the place.

Safeties on the cooling units start to trip as they exceed their load and refrigerant pressures rise.

Your cooling is now N-6.

Your cooling is now N-7.

Your cooling is now 0.

City: just take a walk through manhattan and in a block or two look at the giant open-pit excavation with a 200-year-old morass of undocumented infrastructure under the street. This is before you even try to run fiber up to units in buildings which were built before electricity was standard. I am hardly saying it can't be done, simply that it is not as easy as density makes it seem.

State: the exact opposite problem -- just drive two hours north of NYC and (if you're not still in manhattan) you'll be in some fantastic areas of the state, but, the exact opposite problem exists.

Of note, I do think both of these problems are solvable and we should fundamentally solve them. Just anybody who thinks it's easy or cheap to do so is being myopic. If spent wisely, could be a very useful investment of our money, however.

That's so confusing to me I had to read it five times. Are you saying you lose the metadata, or that the underlying data is actually mangled or gone, or merely that you lose the metadata?

One of the greatest features of something like this to me would be the ability to durable even beyond JuiceFS access to my data in a bad situation. Even if JuiceFS totally messes up, my data is still in S3 (and with versioning etc even if juicefs mangles or deletes my data, still). So odd to design this kind of software and lose this property.

A year or so later I upgraded to a v.92 only to realize my ISP (I think it was IDT at the time) didn't support that and only supported some other 56k "standard" (details are sketchy on this, I was like 12). I was devastated and it was too late to drive back to computer city to exchange it for the correct one.

Now I have 10G symmetric in my house.

There is important additional context around it, of course, which mitigates (should remove) any criminal legal implications, and should also result in google unsuspending his account in a reasonable timeframe but what happened is also reasonable. Google does automated scans of all data uploaded to drive and caught CP images being uploaded (presumably via hashes from something like NCMEC?) and banned the user. Totally reasonable thing. Google should have an appeal process where a reasonable human can look at it and say "oh shit the guy just uploaded 100m AI training images and 7 of them were CP, he's not a pedo, unban him, ask him not to do it again and report this to someone."

The headline frames it like the story was "A developer found CP in AI training data from google and banned him in retaliation for reporting it." Totally disingenuous framing of the situation.

> Sharing encryption key for all team members

you're enrolling a particular users public/key and encrypting a symmetric key using their public key, not generating a single encryption key which you distribute. You can roll the underlying encryption key at any time and git-crypt will work transparently for all users since they get the new symmetric key when they pull (encrypted with their asymmetric key).

> Version control is pointless

git-crypt solves this for local diff operations. for anything web-based like git{hub,lab,tea,coffee} it still sucks.

> - Unless you are really careful, just one time forgetting to encrypt the vault when committing changes means you need to rotate all your secrets.

With git-crypt, if you have gitattributes set correctly (to include a file) and git-crypt is not working correctly or can't encrypt things, it will fail to commit so no risk there.

You can, of course, put secrets in files which you don't chose to encrypt. That is, I suppose, a risk of any solution regardless of in-repo vs out-of-repo encryption.