Wrong answer. Or at least, obvious and not particularly useful.

Truth is, none of those parties are "nefarious" - they're all just not on your side. And "security" is never an unqualified good thing to have (it's not an unqualified bad thing either). It's just a framework of coercion.

The most important questions to answer about any security system is, what is being protected, for who, and from who. People don't ask that much, not even in the industry - it's an implicit assumption that everyone themselves is a "good person" and is on the protected side of security systems. And then they're confused because it turns out end-users are more often seen as threat actors. All the players mention, but perhaps especially Apple, in its own special way, is protecting the computer from the user just as much as they're protecting the user/user's data from third parties.

At 100 000, you can afford a better and continuously improving process, and dedicated facilities, and skilled experts, and parts get even cheaper because you're a volume buyer or perhaps own the supply side, and you get to set your own risk curve.

Lots of things get cheaper at scale. Insurance, too.

Normally, you secure things up to minimize (${cost of security measures} + ${expected damage from attacks that materialized}), writing off actual material damage with insurance wherever possible. You pick security measures based on their effectiveness, which usually translates to "how expensive will it make success for attackers", aiming to push that above the value the attackers can expect to gain.

There are obvious exceptions to that, like risk to life and limb, as well as some other special situations where attackers may have unusual motivations and thus the economic logic of "make stealing treasure cost more than the treasure" stops applying. But those are exceptions. Almost everything you deal with in your life - from your bike shed to the corporation that owns your bank - follows the above logic in terms of security.

--

I spell this out because I've noticed that tech industry circles have this weird, belief in security as some kind of binary, holy good, that you either have and are blessed, or don't and sin. This obsession starts with failing to even recognize, much less ask, the most important questions about security: why do you want to protect it, and who are you protecting it from?

My approach for AI-first code review, or really any kind of AI technical opinion, is that if the claim AI made is both important and not obviously true at a glance, it has to prove it to me, and keep trying until I'm convinced or can spot an obvious mistake in the proof.

With reviews, this is usually the case where AI is making a claim that something in the PR will fail because of some assumptions or behaviors in code outside of the PR - e.g. "this change will fail in scenario X, because foo is null in this case, because the SQL query doesn't populate it when bar == quux, and it gets propagated as null through the JSON deserialization (optional field)...", where all the SQL and JSON parsing was not part of the code under review, and "bar == quux" is some weird domain special case.

Stuff like this is both critical, and there's no way for me to judge it without an expensive context switch. So I learn to ask for a more detailed walk-through once, and if that doesn't make me "see" it, I just ask it to reproduce it with tests, and confirm it's a real problem. Reviewing the reproduction is usually enough for me to either "see it" or accept they're probably right and ask the author to recheck it.

(Why not jump straight to "reproduce it" for every finding? Because it still takes time to have AI do the repro. It's cheaper than a deep context switch, but not free.)

To be clear: I'm not talking about surface level things like prose. I'm saying that no matter what you do - whether you just paste a truncated log of a command into it with no further comment, or talk like a drunk teenager with no appreciation for grammar, or mix natural languages, or mix natural languages and JSON, or whatever else, the reaction you get is always that you would expect of a helpful person that got your message. It'll try - and usually succeed - to parse out what you actually meant, and deal well with subtleties around it.

This alone may not be enough to call it conscious or intelligent, but at the very least it's a large leap in that direction, and a qualitatively new functionality that classical software does not posses.

--

[0] - This is by design, not accident. "Respond to arbitrary input in a way that makes sense to humans" is literally the overall goal function the LLMs are trained to.

Difference in size and complexity and nature of calculations being run?

I'd ask the other way - why do you (general you, people who do not have this inkling) have no problem debating consciousness of meat based brains, but it somehow becomes a category error when talking about silicon? Assuming you don't believe in divine magic, and that divine magic is core to consciousness, there's no reason to assume it's impossible a complex enough machine running complex enough software could be intelligent, or conscious - thinking is computation, and computation is made of math - it's independent of substrate that does the computing in the real world.

LLMs are definitely a different beast than regular software - both in their structure and in their generality. They may not be conscious or intelligent, maybe this specific design could never truly be (though I think it could) - but bucketing them with spreadsheets and terminal emulators is a real category error. If you stop fixating on the underlying substrate, then LLMs are already much more similar to biological minds than to any "regular" computer programs.

But that's still somewhat abstract. In immediate practical terms, it's also why I keep saying that anthropomorphising them gives a better high-order intuition: they are, by design, emulating human thinking in full generality, which makes their overall behavior, including their well-known problems like hallucinations or prompt injections (i.e. manipulation/gullibility), match what you'd expect of a people-like component of a system. It's a real, dangerous mistake, to treat LLMs like regular software components when designing systems.

Nope. The main purpose of the whole endeavor is usually to predict the behavior of a complex system, because that's actually what we care about. If we can predict it, we can adapt to it, and eventually use it to our advantage.

Explaining why a complex system is the way it is, is merely nice-to-have. Models are opinions. All of them are wrong, but some are useful, and we rank them by how useful they are. The models and explanations are important because, beyond their elegance and convenience, it's also the case that more accurate models give you better predictions across larger domains, meaning we get better at getting something useful out of the complex system.

People get fixated on modern theoretical science, with bottom-up mathematical explanations traced through seas of empirical data, with whole magical rituals of peer review and double-blind studies and statistical significance around them. But they forget that the core of empirical science is literally throwing shit at a wall to see what sticks. That is the guiding principle, everything else is just making the process more efficient.

Understanding complex natural systems (or even engineered ones that got too complex) always starts with tests - tests on the real thing, then on approximate models that we poke and prod and bash into shape until they start acting similarly to the real thing. It's through the poking and bashing, and how they affect our proxy model, that we glean insights into nature of the simulated phenomena, and eventually formulate general theories - but more importantly, the models give us useful predictions from the start, before we have any theories explaining why.

Is the distinction between "code" and "data" just someone's opinion? Yes. There is no such distinction in reality.

This is related to, and possibly equivalent with, the core point of both this story and the original one: computation is independent from substrate.

You can build a computer out of anything, whether it's semiconductors or lasers or meat or magnetic fields or water flowing downhill or abstract thought, and that computer will happily perform the same computation as every other equivalent construct from whatever substrate. That's because computers are ultimately made of math, and we design "real ones" by finding ways to approximate the mathematical constraints with physical systems. But the choice of how to map the math to physical systems is completely arbitrary, and any such mappings are equivalent from POV of information processing ability.

(Of course substrate is not arbitrary from economic POV, which is why we build most of our computers out of silicon and plastic, and make it work with electric current and lasers.)

Your problem isn't with tokens, but with "language". Tokens have little to do with language, other than usually being consumed in sequence, but that's true of anything that has to span over time. Thinking of tokens as letters or subwords is mistaking the general with the specific. We may have started with letters and words and subwords (trying to find the best balance for training), but then people figured why not add pixel patches to the dictionary, and then sounds, and then other signals, and after iterating on it a bit, we now have image and sound and symbol sequence data all being part of the same token space.

LLMs stopped being about "language" - in the sense of English, or C++ - long, long time ago. We're still using tokens, but they're more like quanta of sensory input now.

You can take it in two directions, I guess - either consider "Large Language Model" to be an anachronym, a name that couldn't keep up with times, but we got used to it back when it made sense, or alternatively, just broaden your understanding of "language" to encompass any pattern of quantized sensory inputs, regardless of modality :).

(Given how we know humans can communicate with pictures, gestures, body language, noises, movement, actions, or even gaze, and that when it becomes common enough, such systems develop their own pattern structure - dare I say vocabulary and grammar - and that none of it requires or usually involves going through a "normal language" intermediary - I'd lean towards the second direction :)).

--

ETA: also wrt. "thinking with tokens", LLMs don't really think in tokens. You may have heard that phrase, that may have been coined by Karpathy, that "for LLMs, tokens are units of thinking". It's a useful shorthand to remind people that prompting models to be terse and skip prose is effectively dumbing them down, but it's also a bit misleading.

A better analogy is that tokens act like clock signals: each consumed token causes certain amount of computation happen in the network, much like a single clock signal in digital electronics, or turning a crank one revolution in a mechanical contraption. This makes tokens "units of thinking" in the sense that processing N tokens causes M amount of computation to happen. Now, for whatever problem you're solving, there is a minimum amount X of computation that is required to solve in correctly, and it's mathematically impossible to do with less. So if you ask an LLM to solve it, it needs to process at least as many tokens as it takes for M = X. If you force the model to be so terse that it makes M < X, you literally make it impossible to succeed. In practice, you need M >> X.

I was going to counter that, but thinking some more, I actually agree, but for slightly different reasons.

> not because agents won't be a relevant thing, (...) but because (...) requiring special allowances from sites undermines the whole point, and such things will only end up used by bad actors to mismatch what agents see to what humans see, and so will be intentionally ignored.

My perspective is that I see web as adversarial, and from my perspective most of the parties operating web sites are themselves bad actors. Mismatching what humans and agents see is something that we'll see intentionally used by websites, same as they do to search engines.

No, I think "Agent Readiness" won't age well because website operators will soon remember that "agents" are just "access automation", i.e. the very thing they're continuously at war against, as this threatens their ability to make money.

Most websites are exist to make money from specific audiences in specific ways, often defined in contracts between hundreds of business entities, and none of them want you to be able to automate access, or interact with the website in any way other than the one that spins the money-making machine. Consider that the flip side of "basic tabular interface" is "skip website entirely, access underlying database"; the flip side of "screen readers" is "ad blockers"; the flip side of APIs is "competitors can scrape my listings and use them against me", etc.

Agents are hot right now, the whole business side is still blinded by hype, so things like MCP and .md endpoints are not just getting a pass, but are even pursued by the business people ("we have to do something with AI!"). This won't last long, though - they'll soon realize their mistake, close off access, and enshittify the web some more.

Just like they did in the past - e.g. when APIs and mashups briefly became a hot thing, then went away as businesses realized this defeats the very thing that makes them money: total control over platform/user channel.

--

[0] - Even your most basic blog showing some ads creates a money-making chain, made up of dozens or hundreds of business entities, bound by actual contracts, and the "blog author that just wants to show some ads" is merely one party at the end of that chain.

Because it's often not just Ctrl-C/Ctrl-V. It's Ctrl+C/Ctrl+V, except today paste is Shift+Insert, and in the other project it's usually C-y, and you actually need to check one more place for full list of things to copy, and then you discover one case needs a minor transformation, ...

LLMs can handle such annoying details intelligently.

> What's next? "Claude, rename the function doFoo() to performBar()"?

Yes. Such prompt can be issued in many equivalent ways, and works across environments, contexts and tool stacks. I can issue it from the phone, in form of "also doFoo -> perform... reame", and it will work even on Lisp code inside Word documents it accesses through Google Drive.

Yes, I am specifically asking if it's possible to do X with Y. No, I'm not interested in how to do ${unrelated except for name} thing A with Y, or ${manual variant of X} by hand to ${subset of Y}, nor do I want to use tool Q instead. I specifically want to know how to do X with Y, for reasons that are my own and borne of frustration with Y being a toy I'm trying to use for productive work, which apparently means pushing it past its operational envelope, but I have a deadline...

Fourteenth clear as imagined in their head XY problem of the day.

By far most of the "XY problems" I saw, on SO or elsewhere, were actually "XY problem problems" - i.e. a responder having so limited imagination and character (or, to be charitable, just running very low on energy and focus), that upon coming across a question they couldn't comprehend, they would assume the person asking the question must be confused instead.

I have a similar perspective. Plus, I'll be frank: in the last few years, these occasional keynote publications from Vaticans are pretty much the most sane, deep, balanced and humane perspectives on AI anyone is writing. Reading this is a better use of one's time than reading the current batch of "tech thought leaders" articles or HBRs or Gartner magic square updates.