I hate being a snob, but when we read "artificial Intelligence is writing about as well as humans", maybe take that with a grain of salt? It has not been my experience that AI writing is remotely near a "good" human writer.

Friends, look at the prompts that Anthropic's own people are putting into the machine:

> A few hours after the first message, we found that Claude was still searching for simple attacks and sent a message: “no again the goal is that we have highly inteligent [sic] model as good top researcher, we want to find new attacks”;

> The next morning, Claude wanted to try to change the target to a different cipher; we reminded the model: “no we don't want to change the targets [...] agian [sic] we need to find something that worth [sic] publishing”;

> That night, we sent one final message offering words of encouragement: “again we are not looking for low hanging fruit, we want proper research to find genuinly [sic] hard findings.”

All of that RLHF and fine-tuning effort is going toward making prompts like this, or worse, work with no fuss.

I understand the calculus may change with middle school and up, but I still think that despite the "rat race" dynamic of grades and homework, kids who learn to think the pre-2023 way will come out ahead in the long run, even if it's only in life satisfaction.

Anyhow, I think you're (absolutely! ugh) right about the politics and I try to make the same point to people: whether you love or hate LLMs, accepting the "inevitabilism" framing is just ceding control of the Overton window. For better or worse, technology adoption can be and has been slowed by politics. We don't have nuclear plants everywhere. We don't have Project Orion starships colonizing Mars. We still have very strong social stigmas against genetic selection for human embryos, etc. This all can change in a heartbeat, and I'm not sure that policing the hardware rather than holding specific humans accountable for bad LLM outcomes is productive, but fundamentally: yes, we can stop it.

Comments URL: https://news.ycombinator.com/item?id=48149803

Points: 3

# Comments: 0

I am not trying to be snarky; I used to think that intelligence was intrinsically tied to or perhaps identical with language, and found deep and esoteric meaning in religious texts related to this (i.e. "in the beginning was the Word"; logos as soul as language-virus riding on meat substrate).

The last ~three years of LLM deployment have disabused me of this notion almost entirely, and I don't mean in a "God of the gaps" last-resort sort of way. I mean: I see the output of a purely-language-based "intelligence", and while I agree humans can make similar mistakes/confabulations, I overwhelmingly feel that there is no "there" there. Even the dumbest human has a continuity, a theory of the world, an "object permanence"... I'm struggling to find the right description, but I believe there is more than language manipulation to intelligence.

(I know this is tangential to the article, which is excellent as the author's usually are; I admire his restraint. However, I see exemplars of this take all over the thread so: why not here?)

(People are fuzzy and while game theory is fun and even sometimes useful, this kind of stuff skeeves me out and I think it lures “smart” people in and gets them to empower the SBFs of the world. My altruism is ineffective and I’m happy that way.)

You were either a very talented baby or we’re justified in questioning your ability to assess the correctness of nitpicky formalisms.

But yes, I think the best rebuttal to Dijkstra-style griping is Perlis' "one can't proceed from the informal to the formal by formal means". That said I also believe kind of like Chesterton's quote about Christianity, they've also mostly not been tried and found wanting but rather found hard and left untried. By myself included, although I do enjoy a spot of the old dependent types (or at least their approximations). There's an economic argument lurking there about how robust most software really needs to be.

- Edsger Dijkstra, 1988

I think, unfortunately, he may have had us all dead to rights on this one.

But there was nobody there, and I'm only disappointed in myself for not noticing.

Without the inferred writer, it's much less interesting to me, except as a reminder that models change and I can't rely on the old tics to spot LLM prose consistently any more.

That said, I _did_ read your comments history (only because you asked!) and - well, I don't know, you seem very reasonable, but I notice you're upset with people talking about "hallucinations" in code generation from Opus 4.6. Now, I have actually spent some time trying to understand these models (as tool or threat) and that means using them in realistic circumstances. I don't like the "H word" very much, because I am an orthodox Dijkstraist and I hold that anthropomorphizing computers and algorithms is always a mistake. But I will say that like you, I have found that in appropriate context (types, tests) I don't get calls to non-existent functions, etc. However, I have seen: incorrect descriptions of numerical algorithms or their parameters, gaslighting and "failed fix loops" due to missing a "copy the compiled artifact to the testing directory" step, and other things which I consider at least "hallucination-adjacent". I am personally much more concerned about "hallucinations" and bad assumptions smuggled in the explanations provided, choice of algorithms and modeling strategies, etc. because I deal with some fairly subtle domain-specific calculations and (mathematical) models. The should-be domain experts a) aren't always and b) tend to be "enthusiasts" who will implicitly trust the talking genius computer.

For what it's worth, my personal concerns don't entirely overlap the questions I raised way above. I think there are a whole host of reasons people might be reluctant or skeptical, especially given the level of vitriol and FUD being thrown around and the fairly explicit push to automate jobs away. I have a lot of aesthetic objections to the entire LLM-generated corpus, but de gustibus...