Often a mathematician or physicist will use their intuition to speed up the naive brute force of candidate well formed formula variations so that the desired properties emerge, postulating the existence of an intersection on multiple desiderata can in itself be viewed as a novel conjecture, to be proven or disproved.

A very basic (unimpressive) example for an example desideratum is regularity or compactness. the tau=2 * pi substitution does make a whole bunch of expressions more slightly more regular and compact. That is something objective and measurable on a system of theorems.

There is no mathematician's moat vis-a-vis machine learning at a fundamental level. There can be artificially sustained moat, if AI powers limit the distribution of say cryptographic advance capable models, in jurisdictions outside such AI powers, but even that would be expected to be fleeting and temporary...

Image companies A & B wish to merge for example, and claim lowered future consumer prices as a result of the merger. A government can shape this as a bet: proportional to your excess-price-over-prediction is positive, a government can institute a misprediction tax proportional to such excess. This places the prediction effort correctly with the companies instead of the government (if you believe governments were intrinsically better at predicting than companies, you'd be a communist).

But that is precisely what human mathematicians do, prove new theorems by combining ones proven earlier.

I don't see any fundamental difference in functionality between human intellectual contributions vs performant ML ones (LLM or otherwise).

Whenever we listen or read text we are also predicting the near future content.

Just like LLM's we sometimes correctly predict the next token or word, and sometimes incorrectly.

> The "something deeply wrong" part about AI, that even most technology enthusiasts evidently do not seem to grasp, is that it is still fundamentally a statistical model [...]

Imagine someone could pause the universe with a remote control, scroll back in time a little, press play again, and ask a slightly different question, etc.

In such a thought experiment one could also collect the probabilities for a specific human predicting a next word. Implicitly the brain also has a corresponding statistical model, regardless of the construction being visible or hidden. I.e. human intelligence is also fundamentally a statistical model, so the only thing that remains from your claim is that machines for some unmentioned reason don't possess any "real" intelligence or critical thought...

Is it possible that our aversion is simply driven by educational systems collectively and deeply ingraining into populations the idea that intelligence deserves the high costs commanded. Well of course this justifies higher wages towards the higher leadership positions, etc. Now it turns out that intelligence can be dirt cheap. We discover that the fact that "intelligence must be costly so don't question the costs of leadership" was never fundamentally true, so the real anger is this discovery of mismatch between the old claims which served to explain how every society that claimed to order itself and fill positions accordingly with "naturally pre-ordained individuals". Now we are seeing robots exceed average workers, for effectively a grain of rice.

Whenever we reason, we are not neurotically bruteforcing the possibilities (although sometimes we do, like proof by exhaustion), usually we make predictions heuristically of which assumptions or theorems apply or might apply. It's not any different for machine proof assistants...

I'd be happy to revise my opinion if you can demonstrate similar vocal strength on the terminological aspects for those transitions...

You also shifted the goal posts from qualitative to quantitative performance claims. If we ignore that technologies have multiple figures of merit and pretend it's one dimensional, there is a difference between the claim that the machine isn't "printing" vs the machine isn't "printing as well as a human would".

I don't think any of the human printers in the past exceeded the performance levels of current printing technologies, but surely they did exceed the very first machine printers, every technology gets a foot in the door in some niche, and then progressively captures the initially not-yet-automated skills of machine operators.

Would you say an industrial textile weaving machine doesn't weave? At the end of the day its just automation all over again.

Where were these vocal people when the "raster-oriented ink deposition machines" were being called "printers"? The meat or machine brains of future historians will melt because they can't handle ambiguity, a word gaining extra -yet similar- meaning! A word with multiple meanings, unheard of!

Where were these vocal people when people started using software terminology like "executing", "calling", "throwing and catching errors", as if software were human -clownlike sure- but human?

The danger!

A) verifiable proof generation (a limited closed set of axioms and a minimal algorithm like Metamath's are enough to verify correct vs incorrect machine intuition, plus feeding back the rejection/acceptance to adjust the machine intuition)

B) optimizing an illposed problem (how do we define the inaccessible fitness function)

Yes the field of biology or medicine will advance faster, but only because the mathematics and physics will advance faster, any non-axiomatic or non-postulate ramifications actually depend on unknown variables like fitness functions etc. How do you model the environment? That model will necessarily be oversimplified. What makes us believe that a long tail of ever less frequent distinct failure modes (like cancers, cancer is not a single problem) can not be guessed. It would require oracles and crystal balls: if global warming brings malaria mosquito's up north, then "eliminating sickle cell anemia" can actually turn into a death sentence instead of a life-long disease, but perhaps malaria will not come to higher latitudes, it depends on the future!

Wanting to be viewed as trustworthy is not the same thing as having successfully convinced potential users they are.

That's precisely what I discussed: the case where this person fully complies and ceases distribution (but the distro doesn't cease being distributed by others).

With LLM's anyone will be able to disengage such "default wars" in popular software, so there isn't really any border control when LLM's form a large surface area of the border.

At some point cryptography evolves faster than human cryptographers can follow, at that point anyone can only trust cryptographic schemes proposed by their own LLM models, since they will be direct witness to useful responses for apparently unbreakable cryptoschemes with a mathematically known backdoor for the designing LLM.

Why would one power block accept usage of LLM-derived crypto from another power block, if they know it is likely backdoored? The same within a power block: why would one ally financially trust crypto from another ally in the same power block, if they are witness to how easy it will be to request new convincing crypto schemes with a backdoor? The same within a single nation state: why would one bank accept the cryptography proposed by another bank, if they are witness to how easy it would be to ensnare one another with LLM-designed crypto schemes with backdoors.

Its not that humans are incapable of pulling power plugs when the machines start organizing human-on-human violence, it's that no side, no party feels like they can afford a gentleman's agreement to not consult LLM's to find vulnerable hardness assumptions in mathematical cryptography. Perhaps many powerful entities would be more than willing to pull such a plug, but not before others have provably demonstrated pulling the plug,... which is basically waiting until no other human factions exist.

And there's who pays it, and they will pay private companies to compete in order to assure competitive prices for the government.

the 1/f noise at significant power levels is restricted to the lower and lower frequencies, effectively a slowly varying reference ("0") voltage of the amplifier, measuring ADC ground every other sample effectively recalibrates the offset voltage of the amplifier, think of "correlated double sampling".

Effectively measuring in sequence 0V, signal, 0V, signal, ... moves the signal of interest to a higher frequency band, where the 1/f noise is more tame.

When a cliff blocks the way of a vehicle, we don't say "cliffs kill vehicle travel", insteas we just drive around it...