The last person, I think, most clearly, does "owe" you supply-chain security, in the sense that he bears moral (and ought to be made to bear professional) responsibility for any adverse consequences you may suffer from its lack, though in practice he will probably often protest that he couldn't do anything about it because it's not like he is developer. Whether the developer also owes it is a more interesting question, and I think it greatly depends on what attitude he takes towards the evangelist (does he consider him a nuisance who makes implicit promises the developer is uninterested in delivering, or an ally who raises the dev's profile?).

Long ago, I remember seeing a cartoon which involved a tag-team of two people robbing a third, with A pointing a gun at C and saying "give your money to B", while B comments "I'm really just standing here, but I figure it's best if you do as he says". I'm not sure what exact piece of day-to-day politics this was made to comment on (though it was probably some or another flavour of political violence), but it seems somewhat applicable here as well. The lines just become "accept the supply chain, or suffer my public ridicule" and "I'm just providing the software 'as-is', but you probably should do as he says".

It seems to me that a big part of the point of competitive spectator sports is to send, to the spectator, a message along the lines of "this could have been you". It is hard to argue that the ability to throw a 1kg+ discus exceptionally far is otherwise so useful that would justify all the expense of finding and showcasing the outlier. Therefore, the point of the competition stands and falls with whether the spectator buys this message.

When do spectators tend to believe in it? When should they? Arguably, there is a plethora of reasons why the median American spectator looking at a clip of Usain Bolt running could not in any meaningful sense have been him. Yet, somehow, the "could-have-been-me sense" that people are endowed with transcends these reasons and results in men commonly looking at him and getting some of that could-have-been-me sense that gives the sport meaning, and women looking at him and getting much less of it. To solve this, we maintain a separate women's category. The winner there is not as much of an outlier relative to the distribution of the whole population. Most likely, she is still every bit as dissimilar to the spectators as Usain Bolt is. Yet, the women watching, and the ones merely learning about this event happening through osmosis, get their heart warmed by the dubious sense that this could have been them, and perhaps encouraged to try harder and hold more hope for some other pursuit of their own, in a way that they never would have due to Usain Bolt. Would they or would they not get the feeling for a transwoman sprinter? How would we even measure this?

There was some research about it early on that was shared widely and shaped the folklore perception around it, such as the graph in https://static.wixstatic.com/media/be436c_84a7dceb0d834a37b3... from the GPT-4 whitepaper which shows that RLHF destroyed its calibration (ability to accurately estimate the likelihood that its guesses are correct). Of course the field may have moved on in the 2+ years that have passed since then.

It seems to me that the difference between "iterative improvement" as you put it and "close to the identity" (as in the output is close to the input for most of the volume of the input space) as I put it is fairly subtle, anyway. One experiment I would like to see is what happens to the reasoning performance if rather than duplicating the selected layers, they are deleted/skipped entirely. If the layers improve reasoning by iterative improvement, this should make the performance worse; but if they contain a mechanism that degrades reasoning and is not robust against unannealed self-composition, it should make the performance similarly better.

Considering this, I think (again, assuming the benchmarks themselves are sound) the most plausible explanation for the observations is (1) the layers being duplicated are close to the identity function on most inputs; (2) something happened to the model in training (RLHF?) that forcefully degraded its reasoning performance; (3) the mechanism causing the degradation involves the duplicated layers, so their duplication has the effect of breaking the reasoning-degrading mechanism (e.g. by clobbering a "refusal" "circuit" that emerged in post-training).

More concisely, I'm positing that this is an approach that can only ever break things, and rather than boosting reasoning, it is selectively breaking things deleterious to reasoning.

So plastic straw bans (instead of plastic slipper bans, plastic food packaging bans, taxes on plastic clothes fibres...) are what we get. And because the structure of the cause/problem is the same, the language of environmentalism naturally attaches itself and gives form to the vague sense of moral unease surrounding AI. Governments are surely already building tomorrow's tightly integrated thought police drone swarm complexes, but a crusade against those who simulate a zoo of programming weasels in our midst is much easier and morally no less fulfilling.

> jet fuel/steel beams

This debate was carried out sufficiently publicly that I got the sense people actually ran experiments confirming the pro-beam softening/structural failure/whatever case; certainly the "truther" case should have been taken seriously before that, and with decorum always because there is no situation in which any debate in a moderatable forum benefits from playground behaviour.

Often, it seems like this concept of "disinformation" you invoke just serves as a way people give themselves moral license to suspend normal rules of debate conduct in the face of disagreement. Being charitable to your opponents and having to engage with their claims is tiring and difficult, and sometimes they even come better prepared - how much easier if you can just frame dissent as dangerous enemy action and shut it down.

Do you not think that trying to malign your opposition by putting a comical misspelling in their mouths is a bit infantile as a rhetorical tactic? The same thing being done to you would look something like an insinuation that what is being banned is "hurting someone's widdle fee-fees"; surely the discussion here would not benefit if everyone stooped down to that level.

Either way, in what way is this relevant? If the human's labor is not useful at any price point to any entity with money, food or housing, then they presumably will not get paid/given food/housing for it.

Sure, this is not the same as being a human. Does that really mean, as the author seems to believe without argument, that humans need not be afraid that it will usurp their role? In how many contexts is the utility of having a human, if you squint, not just that a human has so far been the best way to "produce the right words in any given situation", that is, to use the meat-bag only in its capacity as a word-bag? In how many more contexts would a really good magic bag of words be better than a human, if it existed, even if the current human is used somewhat differently? The author seems to rest assured that a human (long-distance?) lover will not be replaced by a "bag of words"; why, especially once the bag of words is also ducttaped to a bag of pictures and a bag of sounds?

I can just imagine someone - a horse breeder, or an anthropomorphised horse - dismissing all concerns on the eve of the automotive revolution, talking about how marketers and gullible marks are prone to hippomorphising anything that looks like it can be ridden and some more, and sprinkling some anecdotes about kids riding broomsticks, legends of pegasi and patterns of stars in the sky being interpreted as horses since ancient times.

It occurred to me that this interpretation is applicable here.

[1] https://en.wikipedia.org/wiki/Chick_tract

Nobody's attention span is infinite. I don't doubt I could understand all details of the EU legislative process and keep track of what sort of terrible proposals are underway if I put in the time, but I have a day job, hobbies that are frankly more interesting, and enough national legislation to keep track of.

If you then also say that the outcome is still my responsibility as a voter, then it seems like the logical solution is that I should vote for whatever leave/obstruct-the-EU option is on the menu. I don't understand why I am obliged to surrender either a large and ever-growing slice of my attention or my one-over-400something-million share of sovereignty.