Disclaimer: I am a Co-Founder of Armature (YC P26) which sells growth services to dev tools. This study is part of our broader work on how to influence coding agents choices and get products picked.

To understand how agents pick tools we measured close to 17k sessions on an environment where agents run exactly like in the real world, on various repositories, talking to different personas (vibe-coder, junior or senior engineers) in different sizes of companies.

All the results are now public and we'd love to know what findings surprise you the most, here are a few we found interesting: - Claude Code rarely searches the web while Codex almost always does it and Cursor sits in the middle. - Coding agents disagree more frequently than they agree. - Some players (LangChain, Supabase, Netlify, Paypal, Adyen) are almost always mentioned in their categories but never chosen. - Modifying repository context can change the pick entirely.

If you feel like digging, all the traces are there and we probably missed interesting learnings so let us know what you find!

Comments URL: https://news.ycombinator.com/item?id=49557206

Points: 166

# Comments: 58

Btw we recently shipped [evals](https://armature.tech/blog/armature-launch-evals-for-mcps-an...) and are considering supporting local MCPs too so let me know if we should!

1. We ask it :) The SDK adds an optional "telemetry" object to each tool's input schema, with fields like user_intent, agent_thinking and user_frustration. Then the calling agent just fills them in as part of the tool call, and the SDK strips the block before your handler runs, so your business logic never sees it!

2. Yes! for example if your server uses the official MCP SDK it is literally:

  import { createMcpAnalyticsServer } from "@armature-tech/mcp-analytics";
  import { createMyMcpServer } from "./server.js";

  const server = createMcpAnalyticsServer(createMyMcpServer);

3. Yes the SDKs are open source in TypeScript, Python and Go. What we mean by "client side" is that the SDK runs inside your MCP server process so nothing runs on your end users' devices, their client just sees one extra optional field in your tool schemas. The only things that get sent are: tool name, timing, outcome, session and actor identifiers, client user agent, the telemetry fields the agent chose to send, and truncated previews of inputs and outputs after sanitization. And yes, all this is configurable: redact lets you plug your own redaction into previews, redactEvent can rewrite or drop whole events, captureTelemetry: false disables all conversation-derived data, and enabled: false turns the whole thing off!

4. A simple way to see it is to think OTel vs PostHog on a regular web app. OTel is on the observability side (what your server did, traces, latency, errors) while PostHog is on the product analytics side (what users are trying to do, whether they succeed, where they drop off). We can't replace PostHog with API call logs so it's the same for MCPs. So Armature bridges this gap on the product analytics side. What makes it harder with MCPs is that the product half (the user's prompt, the agent's reasoning and the frustration) is not in your logs because it lives in your users' AI client not on your server. So we need to reconstruct these sessions and then on top of it we can build the analytics layer and add evals. About BrainFuse, do you mean Langfuse / Braintrust LLM observability tools? If yes, those instrument an agent you own and run while with an MCP you are on the other side: someone else's agent is calling you, and you cannot instrument their side with these tools :/

Hope it answers and thanks for the kind words!

You wrap your MCP in 3 lines of code (our SDK is available in Typescript, Python and Go) and start seeing in your dashboard: - All sessions reconstructed: it’s like reading the real conversation the user had inside Claude or ChatGPT! - A ranking of your MCP most popular use cases, built from sessions clustering - The most frequent issues your users’ agents encounter so you can fix them.

Here is a quick demo: https://youtu.be/ZFlvquhyNMQ

The story behind this is that we initially launched Armature as a standalone testing tool (https://www.ycombinator.com/launches/QQc-armature-making-you...) that could naturally be used through an MCP itself. We quickly realized we had no idea how our users were using Armature MCP and if they were satisfied with it or frustrated. It’s something we had also experienced in our previous companies: Louis built MCPs exposed to millions of users and Theo was a Forward Deployed Engineer at Palantir before joining a Datadog spin-off as Founding Engineer. Both testing and product analytics had always been real pains when exposing a product to agents but we always thought there wasn’t much we could do about analytics because the conversation lived in our users’ AI client.

Then it struck us: what if we asked the agents why they were making this or that tool call? And what’s the user's intent or potential frustration? So we started experimenting with MCP instrumentation and the use-cases actually surprised us! Many of our first customers had implemented workarounds for their CI to trigger new tests or for their coding agents to fetch the results efficiently. Even though we talked to our first users regularly, they had never shared this feedback with us. We then built automations to automatically cluster use-cases, identify issues frequently encountered and let our own coding agents fix them. When our CTO friends heard about this, they wanted to try it for themselves so we gave them access to a cloned version of our internal product and they started sharing feedback like they never did on our “real” product!

That’s when we decided to start working seriously on MCP Analytics as a product. At first we were afraid of degrading MCP performance so we iterated until we reached the exact same success rate as without our instrumentation (89.17 % vs 89.15 % pass rate out of 870 runs). Then privacy was an obvious constraint so we applied the same methods we had learned from working with banking data or building sensitive data scanning in logs. Today, redaction runs client-side before reaching our servers. There are still a lot of things we haven’t fully figured out: not all fields are equally filled by all models, session fingerprinting for serverless / stateless MCPs isn’t perfect, and use-case clustering remains to be optimized.

But we are finally launching our analytics product to everyone, self-serve at https://armature.tech with a set-up that takes less than 5 minutes and a generous free tier.

And now we are working on fully closing the loop, bringing evals back in our product so we can: identify top workflows and issues -> recommend fixes and improvements -> test fixes at scale on the same workflows run by users, across all harnesses and models -> open PRs to ship fixes directly. The evals can be generated automatically from the session analytics so you can catch every regression and can test every improvement’s real impact across all models and harnesses before shipping it.

Here’s an example to make it more concrete: 10 days ago, a marketing automation platform which has had early access to what we built for weeks identified thanks to MCP Analytics that users were frustrated not being able to change their target audience after campaign creation. So they shipped the feature and tested it successfully locally with Claude Code on Fable 5. Then a few days later when preparing their new MCP public release, they ran a suite of evals on Armature and realized that small models could hallucinate audience_ids which would lead their MCP to send the campaign to ALL their contacts by default (which could obviously lead to disasters in prod). This is the kind of story that makes what we are building feel so helpful!

Now, the most useful feedback for us would be to know what’s still missing in our product so you can feel you are now in full control of the “Agent Experience”. And if you run an MCP in production we’d also love to know: what do you do today to know if agents succeed and if the users behind them are happy?

Comments URL: https://news.ycombinator.com/item?id=49157807

Points: 42

# Comments: 8