because it is true that, even though we've got some adversarial aspects built into the verification, that's not truly blind from the actor (unless you explicitly design the use of these in that way, which is what I've considered as the better design)

Quick origin story: vr.dev started as a virtual reality project. The domain fit perfectly. The developer adoption did not. Rather than let a good domain go to waste, I pivoted to the other kind of VR: verification and rewards for AI agents.

The problem I kept running into: agents report success but system state tells a different story. The database row is still active. The IMAP sent folder is empty. The tests pass because the agent modified the tests. Real benchmarks put agent success at 12-30%, and even among reported successes a large fraction are procedurally wrong in ways that are hard to catch without actually checking state.

So I built a library of verifiers that check real system state rather than trusting agent self-reports. There are 38 of them across 19 domains right now, organized into three tiers: HARD (deterministic probes against databases, files, APIs, git), SOFT (LLM rubric scoring for things like tone or coherence that don't have a deterministic test), and AGENTIC (verifiers that actively probe the environment via headless browser, IMAP, or shell).

The design decision I'd most like feedback on is the composition model. SOFT scores are gated behind HARD checks, so if the deterministic check fails, the composed score is 0.0 regardless of what the LLM judge says. The idea is to make reward hacking structurally harder rather than just hoping the judge catches it.

MIT licensed, runs locally via pip install vrdev, no dependency on the hosted API which matters if you're using it in a training loop. Full verifier list at https://vr.dev/registry.

Curious whether the HARD/SOFT/AGENTIC taxonomy makes sense to people, whether fail_closed is the right default, and whether anyone has built something similar and run into problems I haven't hit yet.

https://vr.dev https://github.com/vrDotDev/vr-dev https://pypi.org/project/vrdev/

Comments URL: https://news.ycombinator.com/item?id=47322919

Points: 3

# Comments: 2

Comments URL: https://news.ycombinator.com/item?id=44646436

Points: 2

# Comments: 1

Are you really looking for evidence?

Did HN have the spot recently about bars where you drive RC construction equipment around in a sandbox?

It will be interesting to see how peoples' desire to operate physical objects/robots like RC competes with digital experiences as physical robotics continues to improve.

Once a controlled physical object leaves your field of view though, it probably becomes "virtual" in the sense that, sure, you could have a headset on like a drone pilot. But soon graphics and AI-generated experiences will be convincing realities.

Maybe long way of saying, how cool are toy robots about to get?

Exploit (Oxford def.): "make full use of and derive benefit from (a resource)." Contractor (Oxford def.): "a person or company that undertakes a contract to provide materials or labor to perform a service or do a job"

Did Apple mislead these people into thinking their contracts were indefinite or not potentially bound by some end date? Did Apple retract guaranteed equity in the company?

This is literally what the contractor market is for: provide resources in increasing and decreasing amount due to market demands.

I certainly sympathize for the long tenured full time tech employees getting laid off. I really sympathize with the fact that contractors tend to be non-native or immigrant demographics, or out-sourced to other countries. Maybe that's what you were talking about, but seems like an intentionally generaliezd critique of the contracting industry.