Comments URL: https://news.ycombinator.com/item?id=49293267

Points: 4

# Comments: 0

I "trained" the harness on a subset of Terminal-Bench 2.0 tasks while keeping the LLM (local Qwen3.6-35B A3B) frozen. Making LLM inference and the task environments fully deterministic was necessary for clean credit assignment. I learned this the hard way after spending the initial 1 month on experiment noise.

My final results showed that on the full 89-task Terminal-Bench 2.0 suite, the trained harness matched or beat the official Terminus 2 harness for four LLMs that it never collaborated with during training (e.g. GPT-OSS-120B score increased from 18.7% to 36%, while using 55% fewer input tokens per solve). A harness trained only on SWE-bench improved Terminal-Bench scores too. Here's the write-up: https://www.henrypan.com/blog/2026-07-18-harness-training/

I packaged the training loop as a PyTorch-style framework. https://github.com/workofart/harness-training

Comments URL: https://news.ycombinator.com/item?id=49130109

Points: 2

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=49014399

Points: 2

# Comments: 0

Evaluation results show transfer learning (e.g. harness trained on SWE-Bench tasks solving Terminal Bench tasks) and general capability improvements across multiple model families using the same trained harness that even improved over the official Terminus Harness on Terminal Bench 2.0 [1].

Since this was a general problem, I took the chance to create a general PyTorch-like training framework [2]. Right now, you can train with any OpenAI-compatible API for interfacing with the task LLM and train against Terminal-Bench or SWE-Bench tasks, but you can easily extend it to support any task environments.

[1]: https://www.henrypan.com/blog/2026-07-18-harness-training/#e...

[2]: https://www.henrypan.com/blog/2026-07-18-harness-training/#g...

Comments URL: https://news.ycombinator.com/item?id=48994752

Points: 4

# Comments: 0

The idea is simple, the harness is trained once with a frozen task LLM against a given task environment. Then you can then swap out the task LLM to any model and evaluate the "frozen trained harness" with any task LLM on any new task environment.

Since this was a general problem, I took the chance to create a general PyTorch-like training framework. Right now, you can train with any OpenAI-compatible API for interfacing with the task LLM and train against Terminal-Bench or SWE-Bench tasks, but you can easily extend it to support any task environments.

I wrote a blog post (https://www.henrypan.com/blog/2026-07-18-harness-training) on this journey, including (but not limited to):

- results from using this harness training framework to improve general capabilities across many task LLMs to beat Terminal Bench 2.0 (Terminus Harness) and also transfer learnings towards better task-solving abilities in unseen task environments (e.g. harness trained on SWE-Bench tasks solving Terminal Bench tasks)

- how this framework is built

- learnings on what was missing in my initial version of the project (hint: determinism)

Comments URL: https://news.ycombinator.com/item?id=48981124

Points: 3

# Comments: 0

btw, I went down the micrograd path with numpy-primitives all the way to building a PyTorch clone that can pre-train and post-train LLMs (https://github.com/workofart/ml-by-hand). My learning focus was on the math/calculus <-> high-level APIs, instead of efficiency. I'm glad to see more people tackling this problem from different angles.

Comments URL: https://news.ycombinator.com/item?id=48565830

Points: 1

# Comments: 1

I'm also looking into coding harness self-improvement [2]. An inner LLM (raw LLM request) + harness solves coding tasks, an outer agent like Claude or Codex that proposes harness changes. I experimented with many things in the past few months that made me realize this self-improvement thing that everyone is talking about is just an experiment design problem. I wrote about it here [3]. I'm continuing to improve the infra around the self-improvement loop, to increase signal-to-noise ratio per experiment. I'm also generalizing the infra to expand beyond terminal bench tasks and to collect some data across different models (harness-bound vs model-bound).

[1] https://github.com/workofart/ml-by-hand

[2] https://github.com/workofart/harness-experiment

[3] https://www.henrypan.com/blog/2026-05-25-self-improvement-ha...

Comments URL: https://news.ycombinator.com/item?id=48511641

Points: 3

# Comments: 0

[0]: https://www.owenmcgrann.com/p/the-dead-economy-theory

Comments URL: https://news.ycombinator.com/item?id=48303342

Points: 3

# Comments: 0

Turns out there's a lot of parallels to coding-agent customization (e.g. SKILLS.md etc..) too.

I wrote my experience of building such system here, including the successful and failure attempts during the process, and how I approached the self-improvement loop. It's not intended as a benchmark claim but more of a systems/research writeup.

https://www.henrypan.com/blog/2026-05-25-self-improvement-ha...

Comments URL: https://news.ycombinator.com/item?id=48297185

Points: 2

# Comments: 1

1. https://github.com/workofart/ml-by-hand

2. https://github.com/workofart/ml-by-hand/blob/main/examples/s...

3. https://github.com/workofart/ml-by-hand/blob/main/examples/g...

Comments URL: https://news.ycombinator.com/item?id=47406602

Points: 2

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=47389887

Points: 2

# Comments: 0