If that doesn’t worry you, it should.

The current training loop for coding is RL as well - so a departure from human coding patterns is not unexpected (even if departure from human coding structure is unexpected, as that would require development of a new coding language).

The immediate reply to that comment is: "On the internet, no one knows you're an editor." This is a direct play on the famous 1993 New Yorker cartoon: "On the Internet, nobody knows you're a dog." By setting the anecdote in 1987 (a few years before the World Wide Web was publicly available), the commenter is implying that back in the analog days, if a dog wanted to be a writer or an editor, they couldn't hide behind a screen—they had to sit in a smoky London pub and do business face-to-face.

Which makes a lot of sense actually. I would imagine that's what the replier to you thought you meant.

As AI companies start extracting rent from the prompting, one of two things are going to collapse - either the long tail revenue base of low-value inference is going to collapse, because people won't be using Chat GPT to get a recipe if it costs them money or if it is ad-ridden; or the cost of economically-valuable inference is going to go up - and whether it goes up to economically stable positions is a toss-up.

And I say this as an AI enthusiast with <50% probability of a bubble burst in the short term.

This goes beyond assholes on twitter, there’s a whole subculture of techies who don’t understand lower bounds of risk and can’t think about 2nd and 3rd order effects, who will not take the pedal of the metal, regardless of what anyone says…

The more likely scenario is that he was hired for the amazing ability to move fast and break things.

The ai would still be sycophantic even without the pre-prompt. It’s been reinforced to do so, it’s baked in the weights.

I fear the only solution to this is that a climate correcting perverse incentive materialises, such as fusion at scale being more profitable than fossil fuels, but without mass-panic induced traits such that fission has.

If half doesn’t make you leader what does? Maybe you should elaborate your definition of leader? For me it’s “has the highest market share”. And in that definition half is necessarily true.

It’s funny that for PC’s you went for manufacturers (apple is 4th) but for mobile you went for OS (Apple is 2nd). On mobile devices, Apple is 1st, having double market share compared to 2nd place (samsung).

The need to paint Apple as purely a marketing company always fascinated me. Marketing is a big part of who they are though.

[1] https://en.wikipedia.org/wiki/Market_share_of_personal_compu...

Edit: i am not talking evolution of individual agent intelligence, i an talking about evolution of network agency - i agree that evolution of intelligence is infinitesimally unlikely.

I’m not worried about this emerging a superintelligent AI, i am worried it emerges an intelligent and hard to squash botnet

The first group doesn’t care about the narratives, the second group is too focused on the narratives to see the real threat.

Regardless of what you think about the current state of ai intelligence, networking autonomous agents that have evolution ability (due to them being dynamic and able to absorb new skills) and giving them scale that potentially ranges into millions is not a good idea. In the same way that releasing volatile pathogens into dense populations of animals wouldn’t be a good idea, even if the first order effects are not harmful to humans. And even if probability of a mutation that results in a human killing pathogen is miniscule.

Basically the only thing preventing this to become a consistent cybersecurity threat is the intelligence ceiling , of which we are unsure of, and the fact that moltbook can be ddos’d which limits the scale explosion

And when I say intelligence, I don’t mean human intelligence. An amoeba intelligence is dangerous if you supercharge its evolution.

Some people should be more aware that we already have superintelligence on this planet. Humanity is an order of magnitude more intelligent than any individual human (which is why humans today can build quantum computers although no biologically different from apes that were the first homo sapiens who couldn’t use tools.)

EDIT: I was pretty comfortable in the “doom scenarios are years if not decades away” camp before I saw this. I failed to account for human recklesness and stupidity.

We are an organism born out of a molecule with an objective to self replicate with random mutation

Until it kills us all of course.

First of - I don't know what circles you've been around, but I've not been in work collectives where either designers, UX-ers or data scientists try to insert themselves to do things instead of software engineers. If anything, in any collective I worked in, if a software engineer was to say a peep everyone would retreat like there's no tomorrow and thank god that they don't have to deal with it and the software engineer will.

Secondly - I think you are mistaking a structuring and outlining of a process with that being a mandate or an order to follow the process. When I work with software engineers, I expect them to be agile - not to follow an agile process, but to achieve the objectives of the agile manifesto - namely, to iterate ruthlessly, keep an eye on usage signals and lead with MVP's rather than over-design. Good software engineers do that, bad software engineers don't. Ultimately, I don't even judge software engineers by that - I judge them by the ability to produce results.

I think the implication of your thinking is that this is all nonsense because software engineers innately solve data science problems and design thinking problems when appropriate with appropriate methods - to which I'd reply - there's a shocking amount of software engineers who can't do anything with data and are useless in fitting a linear regression to predict something, let alone doing a Fourier transform - to which, presumably, your response would be "No true software engineer is like that". That's great, but it's not true in the real world. Same with design thinking - there's software engineers who just can't solve problems from first principles (but can, say, create a fail-proof CRUD app to automate a business process).

The real world is messy and full of people who can't structure their thoughts, or can't structure them in all domains at the least - and things like design thinking - or generalists who can be thrown at any data problem and produce something (i.e. data scientists) - are useful. They're not the best solution always, sure, and if they start being protective of territory - it's a problem - but in a normal collective that doesn't happen.

Basically - your objection can be boiled down to "generalists are shit, because they impose process on everyone, including people who understand the domain better" - which tells me more about the collectives you've worked in than the nature of those jobs. In every collective I've worked in, generalists are what you throw at an ambiguous problem to produce some results before you get domain specialists in.