A misconfigured prompt or hallucination can cause an agent to navigate to a phishing domain, expose an API key, or confidently claim a task succeeded when it actually clicked a disabled button.

The standard fix right now is "LLM-as-a-judge"—taking a screenshot after the fact and asking GPT-4, "Did this work and is it safe?" That introduces massive latency, burns tokens, and is fundamentally probabilistic.

We built predicate-secure to fix this.

It’s a drop-in Python wrapper that adds a deterministic physics engine to your agent's execution loop.

In 3 to 5 lines of code, without rewriting your agent, it enforces a complete three-phase loop:

Pre-execution authorization:

Before the agent's action hits the OS or browser, it is intercepted and evaluated against a local, fail-closed YAML policy. (e.g., Allow browser.click on button#checkout, Deny fs.read on ~/.ssh/*).

Action execution:

The agent executes the raw Playwright/framework action.

Post-execution verification:

It mathematically diffs the "Before" and "After" states (DOM or system) to prove the action succeeded.

To avoid the "LLM-as-a-judge" trap, the execution of the verification is purely mathematical. We use a local, offline LLM (Qwen 2.5 7B Instruct) strictly to generate the verification predicates based on the state changes (e.g., asserting url_contains('example.com') or element_exists('#success')), and then the runtime evaluates those predicates deterministically in milliseconds.

The DX looks like this:

from predicate_secure import SecureAgent from browser_use import Agent

1. Your existing unverified agent

agent = Agent(task="Buy headphones on Amazon", llm=my_model)

2. Drop-in the Predicate wrapper

secure_agent = SecureAgent( agent=agent, policy="policies/shopping.yaml", mode="strict" )

3. Runs with full Pre- & Post-Execution Verification

secure_agent.run()

We have out-of-the-box adapters for browser-use, LangChain, PydanticAI, OpenClaw, and raw Playwright.

Because we know developers hate giving external SaaS tools access to their agent's context, the entire demo and verification loop runs 100% offline on your local machine (tested on Apple Silicon MPS and CUDA).

For enterprise/production fleets, the pre-execution gate can optionally be offloaded to our open-source Rust sidecar (predicate-authorityd) for <1ms policy evaluations.

The repo is open-source (MIT/Apache 2.0). We put together a complete, offline demo showing the wrapper blocking unauthorized navigation and verifying clicks locally using the Qwen 7B model.

Repo and Demo: https://github.com/PredicateSystems/predicate-secure

Another demo for securing your OpenClaw:

https://github.com/PredicateSystems/predicate-claw

Demo (GIF):

https://github.com/PredicateSystems/predicate-claw/blob/main...

I'd love to hear what the community thinks about deterministic verification vs. probabilistic LLM judges, or answer any questions about the architecture!

Comments URL: https://news.ycombinator.com/item?id=47365599

Points: 1

# Comments: 0

The Interceptor: We hook into OpenClaw's before_tool_call execution loop. The LLM has no idea the security layer exists. The Sidecar Gate: The tool request is routed to the local Rust daemon, which evaluates the intent against a deterministic YAML policy (e.g., blocking rm -rf, allowing fs.read only in ./src). It fails closed by default. The TUI: The daemon ships with a terminal UI to monitor all agent requests, allows, and denies in real-time. I built this in Rust to get strict memory safety with <1ms of latency overhead. It compiles to a static binary and drops into existing projects with zero friction.

Link to GitHub Repo: https://github.com/PredicateSystems/predicate-claw

Demo (GIF): https://github.com/PredicateSystems/predicate-claw/blob/main...

We already use deterministic post-execution verification for our web agents (DOM snapshot diffing, strictly avoiding the 'LLM-as-judge' trap). Next on the roadmap is bringing that same verifiable state-hashing to the OS level. I’d love to hear your thoughts on the architecture and how you're currently handling local agent sandboxing. Note: If you aren't using OpenClaw, our core engine also supports Python frameworks like LangChain and browser-use in 3 lines of code.

You can read the full architecture and see our enterprise fleet management here: https://predicatesystems.ai/docs/vault

Comments URL: https://news.ycombinator.com/item?id=47219794

Points: 2

# Comments: 1

The main difference is that the “tests” are predicates over live browser state and are often proposed alongside the plan on the fly, not written upfront by a developer. But conceptually it’s very close: make the expected outcome explicit, try an action, verify, and only move forward if the condition actually holds.

On “verification wrong”: we try hard to keep predicates grounded and re-evaluated, not “check a cached handle”. Assertions do re-snapshot / re-query during each retry, and we scope them to signals that should change (URL, existence/state of an element, text/value).

If the page is flaky/stale, the assertion just won’t prove the condition within the retry window and we fail with artifacts such as frames of clip (if ffmpeg available) rather than claiming success.

There are still edge cases (virtualized DOM, optimistic UI, async updates), but in those cases the goal is the same: make the failure explicit and debuggable with artifacts and time-travel traces, not silently drift.

The binding between “task/step” and “what to verify” can come from either:

the user (explicit assertions), or the planner/executor proposing a post-condition (e.g. “after clicking checkout, URL contains /checkout and a checkout button exists”).

But the verifier itself is not an AI, by design it’s predicate-only

On the verification side though, dynamic pages are exactly the reason why we scope assertions narrowly (specific predicates, bounded retries using eventually() function) instead of diffing the whole page. If the expected condition can’t be proven within that window, we fail fast rather than guessing.

The verification layer absolutely still exists without it — assertions, predicates, retries, and artifacts all work locally. The API-backed ranking just improves pruning quality on very dense pages, but it’s not required for correctness.

You can set use_api = False in the SnapshotOptions to avoid using the api

The “deterministic overrides” aren’t generated by a verifier agent either; they’re runtime rules that kick in when assertions or ordinality constraints are explicit (e.g. “first result”). The verifier just checks outcomes — it doesn’t invent actions. Because the nature of ai agents is non-deterministic, which we don’t want to introduce to the verification layer (predicate only).

The property-based testing analogy is a good one — once you make success explicit, failures become actionable instead of mysterious.

In practice we’ve seen cases where AX is incomplete, lags hydration, or doesn’t reflect overlays / grouping accurately. It does not support ordinality queries well. That’s why we anchor on post-rendered DOM + geometry and then verify outcomes explicitly, rather than relying on any single representation.

This isn’t about making scripts smarter or replacing Playwright/Selenium. The problem I’m exploring is reliability: how to make agent-driven browser execution fail deterministically and explainably instead of half-working when layouts change.

Concretely, the agent doesn’t just “click and hope”. Each step is gated by explicit post-conditions, similar to how tests assert outcomes:

---- ## Python Code Example:

ready = runtime.assert_( all_of(url_contains("checkout"), exists("role=button")), "checkout_ready", required=True )

----

If the condition isn’t met, the run stops with artifacts instead of drifting forward. Vision models are optional fallbacks, not the primary control signal.

Happy to answer questions about the design tradeoffs or where this approach falls short

This post summarizes four runs of the same task (search → first product → add to cart → checkout on Amazon). The key comparison is Demo 0 (cloud baseline) vs Demo 3 (local autonomy); Demos 1–2 are intermediate controls.

More technical detail (architecture, code excerpts, additional log snippets):

https://www.sentienceapi.com/blog/verification-layer-amazon-...

Demo 0 vs Demo 3:

Demo 0 (cloud, GLM‑4.6 + structured snapshots) success: 1/1 run tokens: 19,956 (~43% reduction vs ~35k estimate) time: ~60,000ms cost: cloud API (varies) vision: not required

Demo 3 (local, DeepSeek R1 planner + Qwen ~3B executor) success: 7/7 steps (re-run) tokens: 11,114 time: 405,740ms cost: $0.00 incremental (local inference) vision: not required

Latency note: the local stack is slower end-to-end here largely because inference runs on local hardware (Mac Studio with M4); the cloud baseline benefits from hosted inference, but has per-token API cost.

Architecture

This worked because we changed the control plane and added a verification loop.

1) Constrain what the model sees (DOM pruning). We don’t feed the entire DOM or screenshots. We collect raw elements, then run a WASM pass to produce a compact “semantic snapshot” (roles/text/geometry) and prune the rest (often on the order of ~95% of nodes).

2) Split reasoning from acting (planner vs executor).

Planner (reasoning): DeepSeek R1 (local) generates step intent + what must be true afterward. Executor (action): Qwen ~3B (local) selects concrete DOM actions like CLICK(id) / TYPE(text). 3) Gate every step with Jest‑style verification. After each action, we assert state changes (URL changed, element exists/doesn’t exist, modal/drawer appeared). If a required assertion fails, the step fails with artifacts and bounded retries.

Minimal shape:

ok = await runtime.check( exists("role=textbox"), label="search_box_visible", required=True, ).eventually(timeout_s=10.0, poll_s=0.25, max_snapshot_attempts=3)

What changed between “agents that look smart” and agents that work Two examples from the logs:

Deterministic override to enforce “first result” intent: “Executor decision … [override] first_product_link -> CLICK(1022)”

Drawer handling that verifies and forces the correct branch: “result: PASS | add_to_cart_verified_after_drawer”

The important point is that these are not post‑hoc analytics. They are inline gates: the system either proves it made progress or it stops and recovers.

Takeaway If you’re trying to make browser agents reliable, the highest‑leverage move isn’t a bigger model. It’s constraining the state space and making success/failure explicit with per-step assertions.

Reliability in agents comes from verification (assertions on structured snapshots), not just scaling model size.

Comments URL: https://news.ycombinator.com/item?id=46790127

Points: 56

# Comments: 19

A traditional scraper/script hard-codes selectors and control flow up front. When the layout changes, it usually breaks at an arbitrary line and you debug it manually.

In this setup, the agent chooses actions at *runtime* from a bounded action space, and the system uses the built-in predicates (e.g. url_changes, drawer_appeared, etc) to verify the outcomes. When it fails, it fails at a specific semantic assertion with artifacts, not a missing selector.

So it’s less “replace scripts” and more “apply test-style verification and recovery to AI-driven decisions instead of static code.”

We used it because it’s a dynamic, hostile UI, but the design goal is a site-agnostic control plane. That’s why the runtime avoids selectors and screenshots and instead operates on pruned semantic snapshots + verification gates.

If the layout changes, the system doesn’t “half-work” — it fails deterministically with artifacts. That’s the behavior we’re optimizing for.

This post summarizes four runs of the same task (search → first product → add to cart → checkout on Amazon). The key comparison is Demo 0 (cloud baseline) vs Demo 3 (local autonomy); Demos 1–2 are intermediate controls.

More technical detail (architecture, code excerpts, additional log snippets):

https://www.sentienceapi.com/blog/verification-layer-amazon-...

Demo 0 vs Demo 3:

Demo 0 (cloud, GLM‑4.6 + structured snapshots) success: 1/1 run tokens: 19,956 (~43% reduction vs ~35k estimate) time: ~60,000ms cost: cloud API (varies) vision: not required

Demo 3 (local, DeepSeek R1 planner + Qwen ~3B executor) success: 7/7 steps (re-run) tokens: 11,114 time: 405,740ms cost: $0.00 incremental (local inference) vision: not required

Latency note: the local stack is slower end-to-end here largely because inference runs on local hardware (Mac Studio with M4); the cloud baseline benefits from hosted inference, but has per-token API cost.

Architecture

This worked because we changed the control plane and added a verification loop.

1) Constrain what the model sees (DOM pruning). We don’t feed the entire DOM or screenshots. We collect raw elements, then run a WASM pass to produce a compact “semantic snapshot” (roles/text/geometry) and prune the rest (often on the order of ~95% of nodes).

2) Split reasoning from acting (planner vs executor).

Planner (reasoning): DeepSeek R1 (local) generates step intent + what must be true afterward. Executor (action): Qwen ~3B (local) selects concrete DOM actions like CLICK(id) / TYPE(text). 3) Gate every step with Jest‑style verification. After each action, we assert state changes (URL changed, element exists/doesn’t exist, modal/drawer appeared). If a required assertion fails, the step fails with artifacts and bounded retries.

Minimal shape:

ok = await runtime.check( exists("role=textbox"), label="search_box_visible", required=True, ).eventually(timeout_s=10.0, poll_s=0.25, max_snapshot_attempts=3)

What changed between “agents that look smart” and agents that work Two examples from the logs:

Deterministic override to enforce “first result” intent: “Executor decision … [override] first_product_link -> CLICK(1022)”

Drawer handling that verifies and forces the correct branch: “result: PASS | add_to_cart_verified_after_drawer”

The important point is that these are not post‑hoc analytics. They are inline gates: the system either proves it made progress or it stops and recovers.

Takeaway If you’re trying to make browser agents reliable, the highest‑leverage move isn’t a bigger model. It’s constraining the state space and making success/failure explicit with per-step assertions.

Reliability in agents comes from verification (assertions on structured snapshots), not just scaling model size.

Comments URL: https://news.ycombinator.com/item?id=46706564

Points: 28

# Comments: 8