SSH pretty much already does this. Per default (using -X) X11 forwarding is in untrusted mode, which makes certain unsafe X11 extensions unavailable. So remote clients already cannot snoop the whole keyboard input.

While the high-speed tracks in Germany are indeed quite a bit of a patch-work, there are over 1000 km of track certified for >= 250 km/h (as of 2015; quite a number of more lines got finished since then, but I could not find the updated number that included them) and by now really rather long corridors are very high-speed. The route from Munich (south of Germany) to Berlin is now mostly covered with upgraded routes for example. I think the 4 hours for that route are quite competitive to Shinkansen times. The fastest Shinkansen route (from the listed operating speed the only one that actually operates at 320 km/h; all others only operate at 260-300 km/h) is the Tōhoku Shinkansen line, which takes 3 hours and 20 minutes for the same distance traveled.

I think it shouldn't be too hard to hack in a dialog to password managers to confirm if the destination is correct before replying to the data request. But even without that one at least notices that a malicious/wrong application grabbed the password.

VNC really sucked on the other hand, not being able to transparently share single windows (at least I never figured out how to), some windows would fail to refresh and I would need to drag them around to get them to redraw, copy and paste was always a pain, sometimes inputs not registering properly.

To be fair to VNC though, over the internet plain X11 forwarding really sucked (latency is the real killer here) and VNC won out there. Unless one was using NX proxy, then it blew VNC out of the water (while using X11 on both ends). Only RDP was somewhat on par with NX over the internet, but locally still beaten by X11.

If you are doing X11 forwarding via SSH it defaults to a more restricted configuration that only allows a more restricted access to the server (no direct sniffing of the input devices for instance).

I think the Qubes approach is the only one worth considering if one deeply cares about security.

In practice I think it doesn't see any adoption, since most people don't run with SELinux or even AppArmor on their desktop and none of the applications run isolated from each other, so it doesn't matter that they all have full access to the X11 server. And for actual security there is qubes, which solves both the application isolation and the X11 security issue.

The problem is the inverse direction (IPv4 -> IPv6) since IPv4 is lacking any mechanism for forward-compatibility.