Hacker News: brammertottens

New comment by brammertottens in "It's Hard to Eval Is a Product Smell"

brammertottens — Sat, 04 Jul 2026 12:06:47 +0000

This is super interesting, and I like the idea of verifiable artifacts that an agent can produce, i.e. notebooks for analysis, links to the source for some claims. Building for scale, it would be interesting to know how the author thinks about automating that and building benchmarks to automate testing the quality

New comment by brammertottens in "CursorBench 3.1"

brammertottens — Sat, 04 Jul 2026 11:32:38 +0000

I mean, they basically did that to make sure that Composer 2.5 sits on the right side, and to a quick reader looks good

New comment by brammertottens in "GLM 5.2 beats Claude in our benchmarks"

brammertottens — Tue, 30 Jun 2026 09:46:05 +0000

This is an interesting finding, but very specialised. It would also be great to get some more information about the benchmark. Is it just a collection of files with vulnerabilities, or are they hidden in a real codebase, where LLM based approaches will not be able to scan every file like a static code scanner is able todo.

New comment by brammertottens in "Evaluating performance and efficiency of the GitHub Copilot agentic harness"

brammertottens — Sat, 27 Jun 2026 06:41:05 +0000

It's an interesting post, but i'm a bit skeptical on their decision to report the best run for each agent, and not just the mean over the 5 runs. We have seen this as well in running benchmarks, that variance within one setup can be pretty big.

New comment by brammertottens in "Show HN: We're inviting Anthropic to put the real Mythos 5 on our open benchmark"

brammertottens — Mon, 15 Jun 2026 10:58:17 +0000

Just a question on the benchmark. It states that it is on real world code, but all the repos in the dataset are intentionally vulnerable repos right, not real world codebases that have reported vulnerabilities?

New comment by brammertottens in "The Speed of Prototyping in the Age of AI"

brammertottens — Mon, 08 Jun 2026 09:06:04 +0000

It goes a bit up and down, compared to two years ago, in my feeling, both anthropic and open ai coding models have made massive jumps. In between big releases I do feel the quality of the models varies over time. 2 years ago I got annoyed with the models being crappy autocompletes, right now I'm just managing them more rather than coding myself

New comment by brammertottens in "The Speed of Prototyping in the Age of AI"

brammertottens — Tue, 02 Jun 2026 06:36:51 +0000

There definitely is the danger of a lot of garbage being shipped, but with both the models getting better and better, and more tools and ways of working being discovered, I believe the quality of what is being outputted is going up as well. Right now, a tool like claude code is more like a junior dev. you really need to steer it well, and know what you are doing, to make sure it doesn't output slop. but with the ability to create agents that work together, implementing design agents, coding agents, review agents, even a non-technical person will be able to design a workflow that spits out quality code, and I think that time isn't far away.