Intelligence has to have a fitness function, predicting best action for optimal outcome.

Unless we let AI come up with its own goal and let it bash its head against reality to achieve that goal then I’m not sure we’ll ever get to a place where we have an intelligence explosion. Even then the only goal we could give that’s general enough for it to require increasing amounts of intelligence is survival.

But there is something going on right now and I believe it’s an efficiency explosion. Where everything you want to know if right at hand and if it’s not fuguring out how to make it right at hand is getting easier and easier.

Comments URL: https://news.ycombinator.com/item?id=46772301

Points: 1

# Comments: 0

Still, getting "something" to compile after a week of work is very different from getting the thing you wanted.

What is being sold, and invested in, is the promise that LLMs can accomplish "large things" unaided.

But they can't, as of yet, they cannot, unless something is happening in one of the SOTA labs that we don't know about.

They can however accomplish small things unaided. However there is an upper bound, at least functionally.

I just wish everyone was on the same page about their abilities and their limitations.

To me they understand conext well (e.g. the task, build a browser doesn't need some huge specification because specifications already exist).

They can write code competently (this is my experience anyway)

They can accomplish small tasks (my experience again, "small" is a really loose definition I know)

They cannot understand context that doesn't exist (they can't magically know what you mean, but they can bring to bear considerable knowledge of pre-existing work and conventions that helps them make good assumptions and the agentic loop prompts them to ask for clarification when needed)

They cannot accomplish large tasks (again my experience)

It seems to me there is something akin to the context window into which a task can fit. They have this compact feature which I suspect is where this limitation lies. Ie a person can't hold an entire browser codebase in their head, but they can create a general top level mapping of the whole thing so they can know where to reach, where areas of improvement are necessary, how things fit together and what has been and what hasn't been implemented. I suspect this compaction doesn't work super well for agents because it is a best effort tacked on feature.

I say all this speculatively, and I am genuinely interested in whether this next level of capability is possible. To me it could go either way.

They marketed as if we were really close to having agents that could build a browser on their own. They rightly deserve the blowback.

This is an issue that is very important because of how much money is being thrown at it, and that effects everyone, not just the "stakeholders". At some point if it does become true that you can ask an agent to build a browser and it actually does, that is very significant.

At this point in time I personally can't predict whether that will happen or not, but the consequences of it happening seem pretty drastic.

I think this is the new turing test. Once it's been passed we will have AGI and all the Sam Altmans of the world will be proven correct. (This isn't a perfect test obviously, but neither was the turing test)

If it fails to pass we will still have what jdthedisciple pointed out

> a non-farmer, is doing professional farmer's work all on his own without prior experience

I am actually curious how many people really believe AGI will happen. Theres alot of talk about it, but when can I ask claude code to build me a browser from scratch and I get a browser from scratch. Or when can I ask claude code to grow corn and claude code grows corn. Never? In 2027? In 2035? In the year 3000?

HN seems rife with strong opinions on this, but does anybody really know?

I think when someone designs a software system, this is the root process, to break a problem into parts that can be manipulated. Humans do this well, and some humans do this surprisingly well. I suspect there is some sort of neurotransmitter reward when parsimony meets function.

Once we can manipulate those parts we tend to reframe the problem as the definition of those parts, the problem ceases to exist and what is left is only the solution.

With coding agents we end up in weird place, one, we have to just give them the problem, or we have to give them the solution. Giving them the solution means that we have to give them more and more details until they arrive at what we want. Giving an agent the problem we never really get the satisfaction of the problem dissolving into the solution.

At some level we have to understand what we want. If we don't we are completely lost.

When the problem changes we need to understand it, orient ourselves to it, find which parts still apply and which need to change and what needs to be added, if we had no part in the solution we are that much further behind in understanding it.

I think this, at an emotional level is what developers are responding to.

Assumptions baked into the article are:

You can keep adding features and Claude will just figure it out, sure, but for whom, and will they understand it.

Performance won't demand you prioritize feature A over feature B.

Security (that you don't understand) will be implemented over feature C, because Claude knows better.

Claude will keep getting more intelligent.

The only assumption I think is right, is that Claude will keep getting better. All the other assumptions require you know WTF you are doing (which we do, but for how long will we know what we are doing).

I'd feel pretty stupid getting worked up about something only to realize that getting worked up about it was used against me.

I'm writing this because for a moment I did get worked up and then had the slow realization it was a phishing attack, slightly before the article got to the point.

Anyways, I think the clickbait is kindof appropriate here because it rather poignantly captures what is going on.

"The energy consumed per text prompt for Gemini Apps has been reduced by 33x over the past 12 months."

My thinking is that if Google can give away LLM usage (which is obviously subsidized) it can't be astronomically expensive, in the realm of what we are paying for ChatGPT. Google has their own TPUs and company culture oriented towards optimizing the energy usage/hardware costs.

I tend to agree with the grandparent on this, LLMs will get cheaper for what we have now level intelligence, and will get more expensive for SOTA models.

Comments URL: https://news.ycombinator.com/item?id=46471071

Points: 4

# Comments: 1

https://www.graalvm.org/latest/security-guide/sandboxing/

That said, it definitely feels as though keeping a coherent picture of what is actually happening is getting harder, which is scary.

Comments URL: https://news.ycombinator.com/item?id=45740555

Points: 3

# Comments: 0

Curious, is anyone training in adversarial simulations? In open world simulations?

I think what humans do is align their own survival instinct with a surrogate activities and then rewrite their internal schema to be successful in said activities.

Right now the state of the world with LLMs is that they try to predict a script in which they are a happy assistant as guided by their alignment phase.

I'm not sure what happens when they start getting trained in simulations to be goal oriented, ie their token generation is based off not what they think should come next but what should come next in order to accomplish a goal. Not sure how far away that is but it is worrying.

I do believe Meta is very bad for the world and has way too much power. Anything that can get people to open their eyes to this is important. Dividing those that are trying isn’t helping.

Comments URL: https://news.ycombinator.com/item?id=17668794

Points: 1

# Comments: 0