I think session transcripts written by coding agents like Claude Code and Codex are very interesting because they offer a detailed window into how work gets shipped. You can see the sequence of decisions that resulted in the final PR, what the agent got wrong, tools used etc. So I built a cli that analyzes these sessions and provides a local dashboard that shows what each session shipped (PRs, features), how much each PR cost, and recommendations for more effective usage.

Concretely, it enriches each session with:

- Outcome links: merged PRs, features shipped, files changed

- Granular cost attribution to outcomes

- Task complexity

- Agent autonomy

- Work type

- Key decisions

- Tool error categories

and across sessions, identifies:

- Agent rework / re-steer themes

- Patterns of deviations from best practices

Combined with the data already in the transcript like model, agent harness and repo, this data lets you answer questions like:

- How much of my AI spend went into PR #2, or feature X?

- Are my agents getting more autonomous over time on complex tasks?

- What's my success rate on repo X vs. repo Y (or any other dimension you care about?)

Works with Claude Code, Codex, OpenCode, and Pi. Everything runs and stays on your machine; enrichments that need an LLM can use your own provider key or a local model.

The repo is at https://github.com/tuneloop/tuneloop, and you can try it by running `npx tuneloop@latest analyze`.

Some things that are in the works next:

- Skill invocation analysis – was re-work needed after a skill invocation?

- harness/model comparisons on merged PRs – creates swe-bench style tasks from merged PRs on internal repos to help compare model/harness choices.

- a version that offers this visibility at the team level.

I’d love to hear from folks here if you find this interesting. Do you look at session transcripts much (perhaps via `/insights` on Claude Code or otherwise)? what do you find useful to track and actionable?

Comments URL: https://news.ycombinator.com/item?id=49112195

Points: 5

# Comments: 0

Seems like teams are encoding procedural knowledge in skills repositories, and I wonder if there’s additional utility from an auto created procedural memory layer

This is interesting, and my prior belief here has been that this automates a one time set up, and perhaps a quarterly clean-up or reactive monitoring changes that people do today. Curious what your experience has been - do teams accept these ongoing maintenance PRs at a good rate?

For full disclosure / context: we work in a related space - investigation agents for production issues.

Curious about this section on the system prompt change: >> After multiple weeks of internal testing and no regressions in the set of evaluations we ran, we felt confident about the change and shipped it alongside Opus 4.7 on April 16. As part of this investigation, we ran more ablations (removing lines from the system prompt to understand the impact of each line) using a broader set of evaluations. One of these evaluations showed a 3% drop for both Opus 4.6 and 4.7. We immediately reverted the prompt as part of the April 20 release.

Curious what helped catch in the later eval vs. initial ones. Was it that the initial testing was online A/B comparison of aggregate metrics, or that the dataset was not broad enough?

Yeah this is a very interesting angle. Our primary mechanism here is via agent created auto-memories today. The agent keeps track of the most useful steps, and more importantly, dead end steps as it executes runbooks. We think this offers a great bridge to suggest runbook updates and keep them current.

>> Curious how much savings do you observe from using runbook versus purely let Claude do the planning at first.

Really depends on runbook quality, so I don't have a straightforward answer. Of course, it's faster and cheaper if you have well defined steps in your runbooks. As an example, `check logs for service frontend, faceted by host_name`, vs. `check logs`. Agent does more exploration in the latter case.

We wrote about the LLM costs of investigating production alerts more generally here, in case helpful: https://relvy.ai/blog/llm-cost-of-ai-sre-investigating-produ...

Most of our deployments are self-hosted, in which case the data stays locally (your chosen LLM provider exempted), if that's what you are asking.

Re: runbooks, yeah even larger teams don’t have good ones to begin with. Relvy helps debug without runbooks as well - it might take longer to explore, but once you are happy with a particular investigation path the AI took, you can save it as a runbook for more deterministic future executions.

Yes, there’s definitely friction there. It may be that the right form factor is that you trigger Relvy’s debugging agent via Claude code / Cursor .

Our early users are heavy on needing to look at the raw data to be able to review the AI RCA, so a standalone set up makes sense. Also, the dominant usage pattern is background agentic execution triggered by alerts, and not manual.

ROI on runbooks (or good documentation in general) is much higher now if you have AI agents running them autonomously in the background. Makes it worth it to write/maintain runbooks.

With the docker images that we offer, in theory, people can re-run the benchmark themselves with our agent. But we should document and make that easier.

At the end of it, you really would have to evaluate on your own production alerts. Hopefully the easy install + set up helps.

Re: diffentiation - yes, faster, more accurate and more consistent. Partially because of better tools and UX, and partially because we anchor on runbooks. On-call engineers can quickly map out that the AI ran so-and-so steps, and here's what it found for each, and here's the time series graph that supports this.

Interesting that you have had great success with Datadog MCP. Do you mainly look at logs?

1) Greater accuracy with our specialized tools: Most MCP tools allow agents to query data, or run *ql queries - this overwhelms context windows given the scale of telemetry data. Raw data is also not great for reasoning - we’ve designed our tools to ensure that models get data in the right format, enriched with statistical summaries, baselines, and correlation data, so LLMs can focus on reasoning.

2) Product UX: You’ll also find that text based outputs from general purpose agents are not sufficient for this task - our notebook UX offers a great way to visualize the underlying data so you can review and build trust with the AI.

A lot of teams are using AI in some form to reduce their on-call burden. You may be pasting logs into Cursor, or using Claude Code with Datadog’s MCP server to help debug. What we’ve seen is that autonomous root cause analysis is a hard problem for AI. This shows up in benchmarks - Claude Opus 4.6 is currently at 36% accuracy on the OpenRCA dataset, in contrast to coding tasks.

There are three main reasons for this: (1) Telemetry data volume can drown the model in noise; (2) Data interpretation / reasoning is enterprise context dependent; (3) On-call is a time-constrained, high-stakes problem, with little room for AI to explore during investigation time. Errors that send the user down the wrong path are not easily forgiven.

At Relvy, we are tackling these problems by building specialized tools for telemetry data analysis. Our tools can detect anomalies and identify problem slices from dense time series data, do log pattern search, and reason about span trees, all without overwhelming the agent context.

Anchoring the agent around runbooks leads to less agentic exploration and more deterministic steps that reflect the most useful steps that an experienced engineer would take. That results in faster analysis, and less cognitive load on engineers to review and understand what the AI did.

How it works: Relvy is installed on a local machine via docker-compose (or via helm charts, or sign up on our cloud), connect your stack (observability and code), create your first runbook and have Relvy investigate a recent alert.

Each investigation is presented as a notebook in our web UI, with data visualizations that help engineers verify and build trust with the AI. From there on, Relvy can be configured to automatically respond to alerts from Slack

Some example runbook steps that Relvy automates: - Check so-and-so dashboard, see if the errors are isolated to a specific shard. - Check if there’s a throughput surge on the APM page, and if so, is it from a few IPs? - Check recent commits to see if anything changed for this endpoint.

You can also configure AWS CLI commands that Relvy can run to automate mitigation actions, with human approval.

A little bit about us - We did YC back in fall 2024. We started our journey experimenting with continuous log monitoring with small language models - that was too slow. We then invested deeply into solving root cause analysis effectively, and our product today is the result of about a year of work with our early customers.

Give us a try today. Happy to hear feedback, or about how you are tackling on-call burden at your company. Appreciate any comments or suggestions!

Comments URL: https://news.ycombinator.com/item?id=47702647

Points: 48

# Comments: 25

Comments URL: https://news.ycombinator.com/item?id=47497668

Points: 3

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=47398668

Points: 6

# Comments: 1

Comments URL: https://news.ycombinator.com/item?id=47339449

Points: 12

# Comments: 0

and execute them when an alert fires.

We are at https://www.relvy.ai

Comments URL: https://news.ycombinator.com/item?id=43848163

Points: 2

# Comments: 0