I embedded a Google Calendar widget on my Book a demo page, I don't know the API and Google doesn't expose/maintain one either.

What we are doing at Retriever AI is to instead reverse engineer the website APIs on the fly and call them directly from within the webpage so that auth/session tokens propoagate for free: https://www.rtrvr.ai/blog/ai-subroutines-zero-token-determin...

All that matters is sociotechnological progress to be able to progress further enough to overcome these tests of existence.

We already have 25k+ users and have an opensource extension as well: https://github.com/rtrvr-ai/rover/tree/main/apps/preview-hel...

Right now since we have a custom sandbox to re-execute the code in, we are using our own syntax and exposed methods. So even now you can edit the generated script.

We are thinking through on self healing mechanisms like falling back to a live web agent and rewriting script.

Since it runs inside your own browser, there should be no captchas or challenges. On failure it can fallback to our regular web agent that can solve captchas.

Big picture wise with the launch of Mythos it might just become impossible for websites to keep up, and they will have to go like Salesforce and just expose APIs for everything.

Compared to your our approach, we are doing this entirely within a browser extension so meeting users where they already doing their existing work.

Within the extension just record doing a task, we reverse engineer the APIs and write a script. Then execute the script from within the webpage so that auth/headers/tokens get automatically added.

You can just prompt to supply parameters and reuse the script at zero token cost.

Use cases we were targetting is like Instagram DMs or LinkedIn connection requests but it should also work for your healthcare use case!

Deeper dive: https://www.rtrvr.ai/blog/ai-subroutines-zero-token-determin...

I really don't want to install an app on to my laptop, especially an MITM, so I think extension would be better.

Our recent technical write up on network discovery/ranking/codegen: https://www.rtrvr.ai/blog/ai-subroutines-zero-token-determin...

The subroutine itself is a deterministic script composed of discovered network calls hitting the site's backend as well as page interactions like click/type/find.

The key architectural decision: the script executes inside the webpage itself, not through a proxy, not in a headless worker, not out of process. The script dispatches requests from the tab's execution context, so auth, CSRF, TLS session, and signed headers get added to all requests and propagate for free. No certificate installation, no TLS fingerprint modification, no separate auth stack to maintain.

During recording, the extension intercepts network requests (MAIN-world fetch/XHR patch + webRequest fallback). We score and trim ~300 requests down to ~5 based on method, timing relative to DOM events, and origin. Volatile GraphQL operation IDs are detected and force a DOM-only fallback before they break silently on the next run.

The generated code combines network calls with DOM actions (click, type, find) in the same function via an rtrvr.* helper namespace. Point the agent at a spreadsheet of 500 rows and with just one LLM call parameters are assigned and 500 Subroutines kicked off.

Key use cases:

- record sending IG DM, then have reusable and callable routine to send DMs at zero token cost

- create routine getting latest products in site catalog, call it to get thousands of products via direct graphql queries

- setup routine to file EHR form based on parameters to the tool, AI infers parameters from current page context and calls tool

- reuse routine daily to sync outbound messages on LinkedIn/Slack/Gmail to a CRM using a MCP server

We see the fundamental reason that browser agents haven't taken off is that for repetitive tasks going through the inference loop is unnecessary. Better to just record once, and get the LLM to generate a script leveraging all the possible ways to interact with a site and the wider web like directly calling backed API's, interacting with the DOM, and calling 3P tools/APIs/MCP servers.

Comments URL: https://news.ycombinator.com/item?id=47810533

Points: 46

# Comments: 17

Compared to your our approach, we are doing this entirely within a browser extension so meeting users where they already doing their existing work.

Within the extension just record doing a task, we reverse engineer the APIs and write a script. Then execute the script from within the webpage so that auth/headers/tokens get automatically added.

You can just prompt to reuse the tools at zero token cost.

Each row spins up a parallel agent, columns mapped as input, agent executes and writes new columns as output.

We tried initial implementation of this with rtrvr.ai building out Sheets Workflows, but I can't help but feel that there is a thread we're pulling towards a deeper insight on this

- agentic visitors to your site can rate, comment, leave notes on your site

- embed our script tag, it will leave instructions on how to call api for leaving feedback as well as track agentic trajectories and identify failures

- more and more of a websites visitors will be agents, but there's no solution to collect metrics and surface analytics on these agents

Deep dive: https://www.rtrvr.ai/blog/roverbook-posthog-for-ai-agents

Comments URL: https://news.ycombinator.com/item?id=47507596

Points: 3

# Comments: 1

On the other hand we construct our own custom Agent Accessibility Trees to represent webpages to models. This approach leads to twice as good performance in WebBench of 300+ tasks (81% vs 40%)

The hard part of raw HTTP scraping was always (1) finding the endpoints and (2) recreating auth. Your browser already has both. We built Vibe Hacking to let the agent use them.

The agent navigates the page, captures network activity, and generates scripts that replay those API calls at scale. Auth propagates automatically because it runs from inside the page.

We tested it on X, it pulled 2,000+ followed profiles despite the UI capping at 50.

DOM-native, no vision/screenshots, #1 on Halluminate WebBench (81.39%). Chrome extension, Gemini Flash Lite default (500 free req/day). Two ex-Google engineers, bootstrapped, 25K+ users.

Happy to answer questions on architecture or limitations.