Comments URL: https://news.ycombinator.com/item?id=50186455

Points: 92

# Comments: 13

This Conductor doesn’t write code or solve tasks directly. It looks at a problem and decides which agents to deploy, what subtask to give each one, and what context to provide. Essentially, it’s not just a router between models — it’s a meta-prompt engineer that assembles a working AI team tailored to a specific task.

What’s most interesting is that this behavior emerged not from hardcoded rules, but through reinforcement learning. For simple questions, the Conductor might rely on a single model call. For complex tasks, it builds a chain on its own: a planner, an executor, a verifier, and a correcting agent. It closely resembles how a strong team breaks down complex work into distinct roles.

The results look impressive. The 7B Conductor was able to outperform every individual model in its pool, including GPT-5, Gemini, Claude, and the open-source models available at the time of the research. The paper reports new state-of-the-art results on LiveCodeBench: 83.9%, and GPQA-Diamond: 87.5%. At the same time, the system proved cheaper than heavyweight multi-agent approaches like Mixture-of-Agents.

One standout feature is called Recursive Test-Time Scaling. The Conductor can select itself as one of the working agents, re-evaluate the output produced by its team, figure out where things went wrong, and assemble a new corrective workflow. In other words, scaling at inference happens not just by “thinking longer,” but by dynamically reconfiguring a new team in response to an error.

The key takeaway here isn’t just that there’s another multi-agent framework. What matters more is this: models are beginning to learn not only how to answer, but how to manage other models. Whereas AI systems used to be built around a single “smartest” agent, the focus is now shifting toward orchestration, roles, verification, and collective reasoning.

And it seems that Sakana is building its new multi-agent system, Sakana Fugu, precisely on this foundation.

Comments URL: https://news.ycombinator.com/item?id=47923668

Points: 2

# Comments: 2

Comments URL: https://news.ycombinator.com/item?id=47850206

Points: 2

# Comments: 0

The dataset was cleaned before analysis: 99% of Informational-severity findings and ~40% of Low-severity were removed, as they consistently lacked sufficient detail to be informative.

The goal was to quantify report quality — not just flag vulnerabilities, but measure how well each one is documented. This became the foundation for a RAG-based audit assistant I've been building, where data quality has an outsized effect on output quality.

Scoring methodology:

Each finding was scored on three primary dimensions — description depth, remediation quality, and presence of a PoC. PoC carried the highest weight, as it is the most reliable signal of a useful report. Solidity snippets and severity level contributed additional points. Raw scores (0–15) were log-normalized to 0–1 to prevent score concentration at the top.

Key findings:

— Total findings analyzed: 23,625 — Mean score: 0.32 | Median: 0.27 — Distribution is multimodal with three distinct quality tiers (~0.05, ~0.25, ~0.60) — ~25% of findings score above 0.51 — these form the high-quality tier ("golden data fund") — All three normality tests confirm the distribution is significantly non-Gaussian

Most counterintuitive result: Critical-severity bugs score lower on average (0.33) than High-severity ones (0.53). Critical findings tend to be reported as brief alerts without PoC — the severity speaks for itself, so the write-up gets less attention. High findings, by contrast, typically include more thorough documentation. This is a problem: the bugs most likely to cause catastrophic losses are often the least well-documented.

What this means in practice:

The three-peak distribution reflects real behavioral patterns in how auditors write reports. The first cluster (scores ~0.05) represents minimal one-liner findings with no context. The second (~0.25) covers standard reports with a description but no PoC. The third (~0.60) is the minority that includes everything: a clear description, remediation steps, and working exploit code. Only this last group is genuinely useful for both AI training and human review.

For the broader ecosystem, the takeaway is uncomfortable: the current standard of audit reporting leaves most findings underexplained. A well-documented bug with a PoC can be understood, reproduced, and fixed in hours. A vague one-liner can stay misunderstood for weeks — or get silently ignored in the next audit cycle.

If you want to see the full distribution charts and statistics for yourself, I put together an interactive notebook with all the visualizations:

https://colab.research.google.com/drive/1Wp4yyEmXYjHATak7Bmy...

Open to questions on methodology or dataset composition.

Comments URL: https://news.ycombinator.com/item?id=47847390

Points: 3

# Comments: 1