What kind of SDK support are you looking for?

The big difference is the complete loop, each PR gets its own VM with the tool chains installed so the agent can run cargo check or cargo tests etc.

We do find the LLMs of today are not the best elite engineers but very very competent junior engineers. It's been a weird but eye opening workflow to use.

ngl the total expenditure was around $10k, in terms of test-time compute we ran upto 20X agents on the same problem to first understand if the bitter lesson paradigm of "scale is the answer" really holds true.

The final submission which we did ran 5X agents and the decider was based on mean average score of the rewards, per problem the cost was around $20

We are going to push this scaling paradigm a bit more, my honest gut feeling is that swe-bench as a benchmark is prime for saturation real soon

1. These problem statements are in the training data for the LLMs

2. Brute-forcing the answer the way we are doing works and we just proved it, so someone is going to take a better stab at it real soon

Comments URL: https://news.ycombinator.com/item?id=42408253

Points: 1

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=42210569

Points: 2

# Comments: 0

I used to think Arc/Rc was a shortcut to avoiding the borrow checker shenanigans, but have evolved that thinking over time.

You do mention it in your comment so wondering if you have anything to share about it

I know my way around this now, which is to literally binary search over the timeline of my edits (commenting out code and then reintroducing it) to see what causes the compiler to trip over (there might be better ways to debug this, and I am all ears)

Most of the times this error is several layers deep in my application so even tho I want to ticket it up, not being able to create a minimal repo for anyone to iterate against feels like a bit of wasted energy on all sides, do let me know if I should change this way of thinking and I can promise myself to start being more proactive.

Having said this, the benefits of borrow checker out weight the shortcomings. I can feel myself writing better code in other languages (I tend to think about the layout and the mutability and lifetimes upfront more now)

My rust code now is very functional, which seems to work best with lifetimes.

I would love to know more about the authors pain, I do hope rustc gets better at lifetime compilation errors cause some of them can be very very gnarly.

I would love to know your workflow, you mention CLI tool or VSCode plugin, which one of them work for you? Whats missing from them where Aide can fill in the gap

We will start including the open file by default in the context very soon (one of the gotchas here, is that the open file could not be related to the question you have)

- looking at git commits - making use of recently accesses files - keyword search

If I set these constraints and allow for maybe around 2 LLM round trips we can get pretty far in terms of performance.

I do wonder what api level access do we get over there as well. For sidecar to run, we need LSP + a web/panel for the ux part (deeper editor layer like undo and redo stack access will also be cool but not totally necessary)

Our first take on solving this is with rollbacks .. which allows you to delete edits up until a point in the conversation.. so if you do notice a bad edit you can do that.

after this, there is the proactive agent ..which checks it's work again and suggests more changes which it needs to do .. you can give feedback and guide it.

With llms we do loose a bit of control but I think the editor should work to solve this