Comments URL: https://news.ycombinator.com/item?id=48712990

Points: 1

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=48711306

Points: 2

# Comments: 1

Comments URL: https://news.ycombinator.com/item?id=48710860

Points: 5

# Comments: 2

I also faced the same problem, so I tried to build something lightweight to stop doing that. Caliper.

It's a local and lightweight harness that runs a skill k times in isolated environments and gives you a pass@k score (How much times it succeeded in these k times). As a non-deterministic technology, you can't just say "it worked once". You need to answer how much it passed in k times.

You define success in a YAML spec. I picked YAML to keep a schema and make it still readable for a human. You either use a LLM judge, a Python assertion, or both:

Here's an simple evaluation example with a JSON extraction, so you write this in a YAML file:

  tasks:
    - name: Extracts action items as clean JSON
      prompt: "Read /tmp/transcript.txt and write the
               action items to /tmp/actions.json."
      expect: "A valid JSON array where every item has
               owner, task, due. No markdown fences."
      assert: |
        import json
        items = json.load(open("/tmp/actions.json"))
        assert isinstance(items, list)
        assert all({"owner","task","due"} <= i.keys()
                   for i in items)

caliper run extract-actions.eval.yaml --k 5 --baseline

What's cool about the --baseline flag is that it will re-runs everything without the skill, so you can see whether the skill is doing the work or the base agent was going to pass anyway:

  ID      Task                           k(5)  pass@k
  task-1  Extracts action items as JSON  5/5   100%  PASS
  With skill   100%
  No skill      60%
  Delta        +40%

I also created two skills you can get started right away with your favorite harness, e.g. Claude Code, Codex or Pi:

- evaluate-skill: run and manage evals without leaving your workflow

- grill-skill: reads your SKILL.md, interviews you about what "good" looks like, writes a 3-task spec (happy path, edge case, adversarial), and runs it

You can install the skill with the command: npx skills@latest add edonadei/caliper

I for now support claude-code, codex, pi, claude-api, openai-api. You can run the agent and the judge as separate backends, so you can run a skill on one and judge with another.

GitHub: https://github.com/edonadei/caliper PyPI: https://pypi.org/project/caliper-eval/

Of course, it's a first step. I think the autorater layer can be vastly improved, more handholding to create and iterate on evaluation specs, supporting more harness, why not including this layer into a self-improvement bigger system?

If you're also building agentic evaluations, I'm genuinely interested to hear how you are handling that.

Comments URL: https://news.ycombinator.com/item?id=48709606

Points: 2

# Comments: 1

Comments URL: https://news.ycombinator.com/item?id=48708941

Points: 311

# Comments: 420

Comments URL: https://news.ycombinator.com/item?id=48708829

Points: 1

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=48707597

Points: 4

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=48707114

Points: 2

# Comments: 1

Comments URL: https://news.ycombinator.com/item?id=48706763

Points: 5

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=48706068

Points: 1

# Comments: 0

Plus, they are not quite optimized for running multiple Claude Code instances, so I created this project to address this issue.

Comments URL: https://news.ycombinator.com/item?id=48705880

Points: 3

# Comments: 4

Comments URL: https://news.ycombinator.com/item?id=48702113

Points: 2

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=48701924

Points: 2

# Comments: 1

Comments URL: https://news.ycombinator.com/item?id=48700895

Points: 2

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=48699921

Points: 5

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=48686649

Points: 2

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=48686082

Points: 1

# Comments: 0

We built what we think the next generation of code review looks like. It's an adversarial, self-healing gate for coding agents that repairs the code on every run, so only clean and secure code moves forward.

Here's a short demo https://www.youtube.com/watch?v=vsq_cHoZCnk

The next agent run also starts smarter than the last thanks to a built-in memory, and because of that agent costs start to stabilize.

It's completely free while in beta. We're looking for testers doing serious agentic engineering to help shape the roadmap.

Install takes about two minutes, and you'll see your first gated review on your next agent run. Give it a spin:

npm install -g @codacy/verity-cli && verity init

Comments URL: https://news.ycombinator.com/item?id=48685853

Points: 4

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=48683595

Points: 2

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=48683357

Points: 4

# Comments: 0