> Giving opencode access to bash and malicious input is not very far from piping it right into bash.

It is very far, obviously. If you have N AUR packages, it needs to send `e` and `:q` N times using the pty tool. You can have it ask you for permission everytime and approve (2N times) (note that when you use yay, you have to press enter N times anyway! so this is just N extra enters but in the opencode UI) or you can even automate an interceptor that checks that it only sends e and :q and no other strings.

> Discovery / creativity

I'm absolutely uninterested in the semantic discussions of what is a real discovery, what is creativity, what is intelligence, etc. I simply don't care. If it's useful great use it. If it's not great don't.

> How small p can be

All that depends on your sampling procedure. If you intentionally smooth the distribution out you can sample the smallest thing, but you pay for it with noise. Taken to an extreme, this is the monkeys typing on the keyboard argument.

It's a mathematical fact that RL cannot improve things it doesn't sample. In any learned distribution you pay a heavy cost by sampling far away from the mode. Most RL algos sample rollouts maybe with some smoothing but that's it. This is why external planners are necessary in order to sample something effectively un-sampleable in the base distribution. Simple example: tool use!

Sutton and everyone are simply calling for a focus on improving these external planners in the same way, as they also enable much better "continual" learning and so on.

> Erdos solution

The RL was what enabled such a huge trajectory to ever become efficiently sampleable in our lifetimes probably. You can do many useful things like this and more purely with the base model distribution.

In fact. Doing RL on user chats and so on especially from pair coding sessions are improving these models coding abilities by a lot making them even more reliable for SWE. In this regard, mode-seeking is a win.

> All sequences are technically in distribution

If it was truly improving 1 in million things systemically, then you wouldn't see base getting the same results given many samples. Albeit they are not erdos problems.

Could it be that at 1T scale, and for difficult problems specifically, grpo somehow filters through the noise and picks out the 1 in trillion? Extremely unlikely (you have your expected rollouts required to sample that, and then you have your sparse reward signal and no credit assignment on top of that...). But of course, only 2 companies in the world can do experiments with it, so there could be some unknown effect the rest of the world has not seen. Barring that, no.

"we have service A that receives a request, it now has a new flag in it, we need you to pass it through to in the call A makes to service B, and then add it in the where clause of the query that B makes".

and expect it to take 2 days including manual testing.

Now you would expect the same much quicker. Any weird bug of the kind "flag not showing up in B because its another weird place where the request _actually_ goes through" that would before suck up 5 hours, would now be found out by the LLM in 2 minutes. "Oh because of this feature being activated in , this new path is used, so you have to add the flag passing logic there". And the next day they get a new task.

This was an extreme example, and it's also not a silver bullet, since now you need to ensure that the intern does the task in a way that they still learn the codebase and the service structure (ideally, they learn quicker) and doesn't become completely beholden to the LLM. So that will also become a skill teams look to hone, how they use tools like this.

Again, no one is saying models can't improve beyond the internet i.e data distribution! They clearly can. The claim is that RL without real exploration cannot exceed the base models distribution, which by virtue of SGD _does_ generalize.

And also, it doesn't mean it's not useful. Improving sample efficiency and making something that happens 1 in 15 times happen 1 in 1.2 times is insanely useful and is what has enabled the kind of coding agents we have today.

Sutton, especially, I doubt has a misconception about this :)

> pass@k

Yeah, AFK now. But it's a well researched thing. You can look for more, but here's one off the top of my head: https://openreview.net/forum?id=4OsgYD7em5 The original deepseek paper also had the result, i.e the paper that first got famous for using grpo as a method that works for LLMs. A side result in one of these papers I forget which one, is that the base model converges in performance with the RLd one at high k.

I'm not saying this makes it useless - it clearly helps for math and coding tasks. But the ceiling exists and that's what the original tweet was referring to. Alpha evolve also shows what lies beyond the ceiling, altho their planner was rudimentary.

Comments URL: https://news.ycombinator.com/item?id=48482039

Points: 3

# Comments: 1

i.e, evaluation, retention yes. variation or "planning" no.

That is not to say you cannot use LLMs. Alpha evolve does exactly that. It uses an external simple evolutionary planner though. The overarching point he's making is that our planner is still "dumb" and we need to work on it.

When you iteratively guide an LLM in claude code, you are the external planner. That also works.

i.e, evaluation, retention yes. variation or "planning" no.

That is not to say you cannot use LLMs. Alpha evolve does exactly that. It uses an external simple evolutionary planner. The overarching point he's making is that our planner is still "dumb" and we need to work on it.

When you iteratively guide an LLM in claude code, you are the external planner. That also works.

In larger models, these fine tuning techniques work more reliably/robustly. Because of this many usecases tend to prefer larger models. It is possible to work the same behaviour into the smaller model, but it requires more effort. But it's one-time. And smaller models are usually much cheaper. People make a tradeoff along this curve.

This is observed at few-B scale upto hundred-B scale. No way for us non-anthropic/openai to fine tune beyond that of course.

No, for static even padding bytes are zero.

For automatic, yes it may effectively turn a = {} to a.member = 0, leaving the padding bytes uninitialised. Or on copies like a = b it may not copy padding bytes.

Yeah that's the issue isn't it? I see in the UI it's sent. But actually it's sent only the next morning.

To be fair. It's fine for an issue tracker. Anything actually important i'd spend a few seconds going over what I just sent. In which case I'd see it's not synced. And what's not that important it's really fine if in some random wifi edge case it's phantom sent. So makes sense.

It also allows you to use it with win-drag of course.