The developer here. My background is in networking/systems and got interested in MechInterp for various reasons, and one of the first things I noticed was many methods that have been built to monitor internal AI state are generally evaluated on how they perform as a 'classifier' of sorts; the idea is to look at the ROC curve or the AUROC on a test set etc. However, being a good 'classifier' doesn't really tell you if the monitor helps you take better decisions, specially under operational constraints. For instance, there is always a budget (time/compute etc) that is imposed on inspecting every operation an agent proposes, which means mistakes will be made and not all mistakes are the same. Take a simple example; an AI agent/assistant wants to read a private log (it's authorized) and deleting a backup. Your audit budget lets you review only one. An internal monitor inspecting the activation of the model may give a 99% chance of violation for reading the log, but a 75% violation chance for backup deletion. If your controller focuses on the higher score, and if you can monitor only one action, you'll miss the more serious deletion action.

This is where ObserverBench comes in. Borrowing from control theory, an observer is a component that estimates the hidden state of a system so a controller can act on the estimate. ObserverBench does this for internal AI monitors: know whether an estimate is useful for a particular decision, by testing it against the actions the decision produces. So instead of just using a loss function, we also use a controller and a budget to evaluate monitors/observers.

There are multiple results on the page, but to highlight one set of results on testing monitors on Gemma 2-9B-it checking code for backdoors, we see: * A prompted monitor (asking Gemma to review the code itself) beats an internal Gemma scope SAE feature probe on ranking (aurora 0.893 vs 0.87) * However, on simulated examples (2000), with 1% backdoor prevalence and 2% review budget, the internal probe left fewer backdoors unreviewed. * Resampling makes the ranking uncertain, but this shows the core issue- a higher test set score (auroc) does not mean you catch more violations under a budget.

ObserverBench fixes the task, actions, budget and decision rule so that one can compare different observers on raw estimates/scores and their downstream decisions.

Two ways to check it out:

* Interactive walk thru (browser, no login): https://kwisatzh.github.io/observerbench/try/ A simple example in a sandbox * Local runner (no GPU needed or model downloads): https://github.com/kwisatzh/observerbench#sixty-second-demo Uses saved measurements from Qwen2.5-7B base to test which internal model edit to choose.

Grateful to get any feedback, curious to know if others running safety monitors have seen this mismatch between eval metrics and budget constrained decisions.

Comments URL: https://news.ycombinator.com/item?id=49645703

Points: 4

# Comments: 1

Author here. Like to present a project I've been working on - Otelcol-GenAI-Sketches, a Otel connector that works on genai/agent traces, and summarizes them (via sketches), to produce:

() exact request counts/totals for tokens; () distinct users/prompts (estimates) (*) 'heavy' items in terms of tokens

Why does this exist: I'm a data scientist by profession, and needed to look at Otel traces, but soon ran into cardinality and portability issues, along with privacy issues (raw prompts being transported elsewhere, like for analysis in a notebook env is a no-no). I wrote a sketchlib first (llm-sketchkit) and then the connector for Otel collectors.

To try it out, just:

(i) git clone https://github.com/llm-measurement/otelcol-genai-sketches.gi... (ii) cd otelcol-genai-sketches (iii) export GENAI_SKETCH_SECRET="$(openssl rand -hex 32)" (iv) make example-up (v) profit!

This would start a demo app that includes a Otel collector, prometheus server/db and a grafana dashboard.

The repos contain benchmarks, accuracy results and some hopefully helpful answers to the obvious question -- why not just use Datasketches?!

Either way, will be grateful if you can try it out, specially on your workloads, and give me any/all feedback.

Thanks for reading!

Comments URL: https://news.ycombinator.com/item?id=49389744

Points: 4

# Comments: 2