I've been building something similar, basically a way to run your full dev environment on your Mac and connect to it from iOS. terminal, files, AI agent all talking to the same session. the tricky part is honestly just keeping state in sync when you switch devices.

like a year ago running an agent on a non-trivial task was painful and expensive. now it's annoying and somewhat expensive. tbh at the rate efficiency is improving I wouldn't bet on "still more expensive than a human" holding for long.

the failure mode I'd worry about most is cascading context drift, where each agent in the chain slightly misunderstands the task and by the time you get to the test agent it's validating the wrong thing entirely. fwiw I think the LanceDB memory is the right call for this kind of setup, keeping shared context grounded is probably what prevents most of those drift issues.

That said I think spzb's point is real, the signaling server is still a trust boundary even if the p2p channel itself is encrypted. Tailscale sidesteps a lot of that by having a more battle-tested auth model tbh.

On the "why would you even want this on phone" thing though, I guess I disagree a bit, I've been building something related: Xtro, terminal + source control + AI agent on iPhone connected to Mac, and like the use case is surprisingly legit once the layout is actually designed for mobile. Emergency deploys, quick fixes, that kind of thing.

Fwiw the janky `claude -p` approach you described is actually pretty solid once you stop fighting it, the simplicity is the feature I think.