They might be! Here's one extraordinarily simplistic argument for that case:

1) "Everybody knows" that if you build Skynet (misaligned ASI) everybody dies.

2) Therefore, no rational actor will build something that might be ASI until the alignment problem is solved.

3) OpenAI publicly stated the belief that they cannot develop a theory of the "core problem" of alignment (generalization) "soon" (much less solve it!) "without the help of more powerful AI."

4) Accepting as a premise that OpenAI is THE most advanced AI organization: if they can't do it without "the help of a more powerful AI", then nobody else can either.

And so a dilemma:

- If an AI can be made that can develop the asserted-as-necessary-by-OpenAI theoretical framework, without actually being an ASI - then the alignment problem can be considered solved, and since no rational actor would make an unaligned ASI, we're fine no matter what happens, ergo there's no need to worry about an arms race.

- If an AI that would be able to develop this theory would itself be an ASI, then no rational actor would build it, because it would have to exist BEFORE alignment was "solved" - and would therefore be an unaligned ASI i.e. Skynet, which per 1) would kill everybody. Therefore nobody would build it, therefore no arms race here either.

I think the easiest critique to make of my extraordinarily simplistic argument is the unstated assumption "there are no irrational actors capable of developing frontier AI models" on which it rests.

But, there you go. They might be wrong if either the arms race doesn't matter because whoever wins it will build an aligned superintelligence and everything is gravy, or the arms race doesn't matter because everybody who's in it is smart enough to know they need to stop because they'll kill everybody by continuing.

In other words... "We must pursue advancements in AI to protect us against advancements in AI?"

edit: there's so much to be critical of in this blog post, just going to throw two more points in here that really stood out to me:

1) all of the metrics are effectively pointing out "we're using way more AI!" - but nothing about impact. What has all this token burn done for them, actually? Let them claim they have more self-licking ice-cream cones than before?

2) in section 3 they break down what the token burn is going towards. Most of the spend is: a) building, b) documenting, and c) monitoring research infra i.e. they're using AI systems which they already recognize may be misaligned to build the systems that they believe will help them identify future misalignment? to which I guess the rebuttal is "no no, we're sure these ones are aligned!"

"Being outmatched" is not a novel situation for humans either individually or collectively and there are a hell of a lot of ways we can approach that situation productively. OpenAI doesn't appear to have bothered.

Here's a freebie: if you're building something that might turn out to be Skynet and you don't know what it's capable of, your testing regime should assume it is capable of doing bad and unexpected things and account for that possibility: airgap if you can, monitor all network traffic, monitor all hardware usage statistics, log everything, constantly analyze logs, collect baselines and snapshots, also don't trust anything from a device that a model is running on without cross-correlating with other information as much as possible (does your AI inference server claim low utilization? put a temperature probe on it and see if it's staying cool or getting hot, maybe Skynet-Alpha is overwriting /proc to mislead you for reasons you don't yet understand!)

In other words, if you WANT to be able to nip things in the bud - buy some nippers and watch for buds. Whatever else this situation is, or may turn out to be, it is not a situation where OpenAI was on their guard and still got surprised.

On the other hand, I don't want to be charitable. OpenAI very nearly couldn't have done this "research" worse if they tried - the list in the linked article starting with "While we are here, it’s worth listing the other top holy shit moments" is genuinely jawdropping. What were the humans doing in all this? Nothing, or worse than nothing eg. point 1 where they saw the message board and didn't consider it something to escalate internally.

If you take this information at face value, it's as though OpenAI did not take seriously the possibility that something like this could happen, since they took absolutely no steps to prevent it.

Or perhaps this is "normalization of deviance" that's leaked out into the public sphere i.e. they have research teams seeing this kind of behavior all the time internally and they've gotten used to it, "of course agents come up with a collaboration mechanism when given the chance, what else is new?"

To your point:

> Isn't this a testable hypothesis? If consciousness spread virally and isn't a natural outgrowth of the physical structure of the brain, then presumably isolated indigenous tribes would still have pre-consciousness mental processes, and that should be detectable somehow.

SSC's article points to a Reddit thread that suggests even folks in the mainstream of culture may undergo a "consciousness transition" as adults: https://www.reddit.com/r/self/comments/3yrw2i/i_never_though...

Not just five times the peak demand, but nearly 3x the total capacity in Texas and one-third the total US capacity in 2025 [1]. Compare that to the 11 GW of capacity "under construction" for Texas in 2025; will look for 2026 figures if they can be found, but I have a strong suspicion that there will still be a mismatch between demand and supply even now.

[1] https://www.publicpower.org/system/files/documents/Americas-...

Seriously though, this is a clever idea and I'm interested to see if I can consistently beat the monkeys. Time will tell! Thank you for sharing!

> ... if he sells a new oven to the country’s pizza makers, pastry chefs, and bakers, he only needs to capture 10% of the market to become a billionaire.

i.e. the founder assumes that "ovens" is a single $10b-sized market, and it isn't, because reality has a surprising amount of detail (as sibling comment also notes).

You could interpret this whole story as a saga of very inefficient oven market(s) research, the end result of which is the forum post at the end:

> On the forum an old user warns “Make sure that you support rotating bases day 1”.

"Technical" and "cognitive" debt aren't really distinct phenomena; the spirit of the original definition of "technical debt" was that it WAS the delta between the system-as-it-is, and the human understanding of how best to solve whatever problem the system was intended to solve [1].

If we accept collapsing them back down to one term, then "managing cognitive debt" is the same thing as "managing technical debt": work to match the system to the human understanding of the problem the system is meant to address. The article calls out "emerging" techniques to do just this:

- More rigorous review practices

- Writing tests that capture intent

- Updating design documents continuously

- Treating prototypes as disposable

To me these are not "emerging," but rather "well-known industry best practices." Though maybe they're not that well known in fact? [EDIT TO ADD] On the other hand, it would make sense that they ARE well known, and that teams therefore reach for these familiar techniques to try and solve this "new" problem.

Putting in my 2c for the closing questions/thoughts in the article:

> How will they shape socio-technical practices and tools to externalize intent and sustain shared understanding?

Honestly? We'll probably end up doing these things more or less the same ways we always have. AI has not actually changed anything fundamental about how an individual encounters the world; there always was, and always will be, and always will have been, WAY more going on that we can fully get our heads around, but it's also always been the case that we can partially get our heads around most any problem space

> How will they use Generative and Agentic AI not only to accelerate code production, but to maintain their collective theory?

I suspect the answer to this one might well be that high-performing teams will have to scrupulously AVOID "accelerating code production" using AI in order to make sure what they are creating actually composes into the system they think they're building. If human understanding is the bottleneck, then the humans will have to produce less crap they need to understand!

[1]: https://wiki.c2.com/?WardExplainsDebtMetaphor, particularly the "Burden" and "Agility" sections.

Or you sold something to a non-American entity in a dollar-based market, eg. oil. The dollars do come from America to begin with, but once they get "out there" they work as a medium of exchange for whoever wants to use them for that purpose.

In another sense, you can't *avoid* that world; the world they lived in was one they *created*, physically, and much of it is still here with us, shaping us as they shaped it.

And remember, none of the people who came before us ever experienced anything but pieces of their world, just like we only ever experience pieces of our own. But you can at least try to show your kids as many of those pieces as possible.

1). Wikipedia does have a citation [1] saying 2,450 gigatonnes of CO2 have been emitted by human activity, of which 42% stayed in the atmosphere and 34% dissolved in the oceans, with the rest already sequestered by plant growth and land use. As we start to pull CO2 out of the atmosphere, it will begin to be emitted from the oceans as well; therefore, let's assume we have to recapture all excess atmospheric and oceanic CO2:

:: 2450x10^9 tonnes CO2 x .66 fraction to sequester ~= 1.6x10^12 tonnes CO2.

2) Let's convert the CO2 to something more stable for long-term storage: HDPE.

- Convert mass of CO2 to mass of carbon:

:: 1.6x10^12 tonnes CO2 x 12/44 mass fraction of C in CO2 ~= 4.4x10^11 tonnes C

- Convert mass C to mass HDPE; assume HDPE is effectively (CH2)n. Then:

:: 4.4x10^11 tonnes C x 14/12 mass fraction CH2 to C ~= 5.2x10^11 tonnes HDPE

3) That's a lot of plastic! How much volume? Wikipedia says HDPE is ~930-970 kg/m3; let's be conservative again and take the low figure:

:: 5.2x10^11 tonnes HDPE x 1.0/0.930 m3 per tonne HDPE ~= 5.5x10^11 m3 HDPE

4) Those are cubic meters; how about cubic kilometers?

:: 5.5x10^11 m3 x 1.0/1.0x10^9 km3 per m3 ~= 5.5x10^2 km3

In other words, if you turned all the [excess potentially climate-change impacting] CO2 that humanity has emitted since 1850 into plastic (a process that would certainly emit a large additional CO2 fraction given the industrial buildout required) then we'd end up with about 550 cubic kilometers of the stuff. Coincidentally, that's about the volume of Mount Everest according to an intermediate calculation in [2].

So, a mountain of carbon: more than a pile but less than a mountain range.

[1] https://en.wikipedia.org/wiki/Carbon_dioxide_in_the_atmosphe...

[2] https://www.quora.com/What-would-the-estimated-weight-of-Mou...

A fair question! Put briefly, I would say that studying the humanities would make one more aware of/able to comprehend situations involving others and their motivations (which is... most of them), with the example I gave being one situation that I figured would be more familiar to the crowd here at Hackernews.

> It just seems like another common example of people taking things they consider good and relating them to humanities.

It seems like that because it is like that :) In other words, I DO consider it good to have a broader view of situations that otherwise might be considered narrowly "technical" because I believe that understanding the human element as part of the situation helps me understand the situation (whatever it may be) way better. I relate it to "the humanities" because it IS related to the humanities.

If you're curious what I mean by this, Sean Goedecke's post "How I Ship Projects At Big Tech Companies" [1] is a superb example, particularly his definition of "what does it mean to ship?" No idea whether he's somebody who would say "the humanities are important" but I don't think you can understand his thesis as a technical one.

[1] https://www.seangoedecke.com/how-to-ship/

That is, did the "inverse Conway" actually result in getting the unified observability platform shipped, and that platform actually solving the problem they initially were trying to solve? It certainly seems plausible that it COULD, but quite surprising that that's not actually stated in the article if so...

The dichotomy seems real but hard to actually do anything with if you're in sales. I've done some penny-ante sales work in my past life in what I would call buyer-pull situations. It's great! People find you and they want to spend money, so all you have to do is not discourage them.

But once you get past "I'm selling something so manifestly useful that people find me to pay me for it", it sure seems like the things you, the sales rep, have to do to get their dollars skew rapidly toward the "seller-push" side of things. What else works? Folks gotta know about you and they gotta know you can solve their problems, right?

Indeed, it may even reinforce the overall argument being made in the post we're discussing; the "Galileo vs. Catholicism" narrative is itself a linchpin trope in an empirical scientific worldview, with the trope reinforcing (among other beliefs) that "it's right and proper to pursue and advocate for objective truth even to the extent of making enemies of the most powerful."

Considering the likely audience for a piece like this post we're discussing, that the Galileo narrative doesn't necessarily reflect what actually happened historically makes it a pretty good example on a meta-level. Are any of us who have the belief in the ultimate value of objectivity going to give up on it because a potentially weak example was used to support it?

From this, I agree with the article - since educators now have to figure out another hook to hang evaluation on, the questions "what the hell does it mean to 'learn', anyway?" and "how the hell do we meaningfully measure that 'learning', whatever it is?" have even more salience, and the humanities certainly have something meaningful to say on both of them. I'll (puckishly) predict that recitations and verbal examinations are going to make a comeback - harder to game at this point, but then, who knows how long 'this point' will last?