And like, it does this despite it speaking in extremely dense math, which both makes it sound correct and requires a lot more effort to prove when it is wrong... yet, it isn't actually correct more often, and so that time sink just isn't worth the benefit. I then think many people--including people who can speak math (as can I)--just stop bothering to check everything, as if you come across a human who speaks like this it probably does correlate with slow and careful thought that helps prevent errors.

Instead, Claude has the mistake rate of a somewhat accelerated beginner impossibly combined with the language of an expert professor; and we as humans just aren't good at that combination: it becomes very dangerous and makes it take longer to spot its egregious mistakes and trained-in biases. If you have to use Claude, I thereby claim you really need to have a team of not-Claudes to help insulate you from this, and Gemini (while being a bit senile) is a lot more collaborative and approaches problems in ways that makes it harder to get tricked.

(To translate this into more of an engineering analogy: Claude always feels to me like the engineer who put more effort into learning how to program in functional languages than into how to actually develop working code, and then confidently presents you answers in Haskell or Lisp that never quite work. To find their errors is then very costly. In contrast, Gemini feels more like a Java or Go developer who knows they are a cog... that's helpful! <- Which maybe just goes to show that AI has finally turned me into a manager, omg.)

Instead, they are focused on stuff like "can I ask the AI to help me build a nuclear bomb" or "is the AI willing to generate pornographic stories", which is neither trying to protect us from unleashing a vengeful god NOR preventing (in any honest way) the today-level problems you (very correctly) bring up.

I do, in fact, think that, if someone says "my technology -- call it what you will, and I won't bother to tell you what it is -- is so dangerous that in the wrong hands or if it isn't carefully contained in a way that I am going to great lengths to try to accomplish it can end the world" -- which is what these people at these companies are saying -- that it would be hard-pressed to not be willing to call that a "doomsday device". I would even go so far as to claim that the exact opposite is more likely true: that to not want to do so requires you to have some extremely awkward bias yourself.

Sure: me using that label to ascribe something related to this product makes some people, seemingly you included, feel bad, and so maybe we can say it involves a "bias", as I clearly can avoid that label, but chose not to, and in so doing colored the discussion in a way that makes you unhappy... but it doesn't make it untrue anymore than any other purely descriptive term is now-a-days often said to be biased. As they like to point out: "reality has a liberal bias" (even if it is just as often liberals annoyed with the terms being used by people ;P).

Again: the bias against the use of that term seems much more fascinating than being upset at the idea that the parable -- which of course is trying to make a point, and so is inherently biased towards that point -- could be attempting to temporary apply it, particularly so as it seems to lead to blindness to what actually happened: people will hear you saying your tech is dangerous, and it will be banned and regulated.

And that's related to why I maintain that I think your example disproves your own attempt at a point: you try to draw a parallel to another technology, as if once we find out it is nuclear reaction research we of course know that that isn't a doomsday device... and yet that is one of the few things that is most obviously to the most people a doomsday device, it is a thing that even the people working on it have always felt might not just be a doomsday device but might be "the" doomsday device (as in, the one that actually does us in), and there has not only been countless people out there worried about what can and will happen in its usage, but it was in fact weaponized and a very very large number of people died due to it, so to try to make the argument that it would be biased to call it a "doomsday device" itself feels very awkward.

But, sure: we can decide that that's biased and throw out the "dunno it sounds like a doomsday device and should be regulated" argument as the argument itself has a bias in it, and that leaves us with... what? A world in which people make these things, talk about how dangerous they are, and then feign in shock and frantically try to backpedal when governments step in to prevent it from being released without careful analysis and regulation and study and planning?

https://chatgpt.com/share/6a2d8c75-56f4-83e8-a61a-301e4c62b1...

Even if this is just temporary, your #3 is more in direct conflict with #2 than you seem to be willing to admit: if you were to own stock in a company that you know has a powerful product and a market lead, but they have been required to take a time out in the market for a year, that should be devastating for their valuation.