We found that AI usage is basically guaranteed now, but certain challenge designs did thwart it. Challenges built with temporal visual elements made AI fall flat on its face, as it could not ingest/process the data fast enough to act on them in time. We also found that counterfactual challenges (ie. the result you get did not match what we suggested you'd get) made AI-assisted solve time slower compared to pure humans, indirectly penalizing over-reliance on AI. Multimodal challenges combining audio and visual elements were also very effective, but were not as accessible to players.

This paper gave us some ideas about designing those challenges: https://arxiv.org/pdf/2308.02950.

For our next event we figured out a way to thwart AI in our CTF: embed the CTF in a game engine. The loop essentially becomes something like this: Connect to a simulated access point in the game, the K8s cluster connects their attack container to a private network with the challenge box(es). Hacking the boxes doesn't render a flag, but rather changes in game state. AI did very poorly coping with this in our testing, as it can't derive the spatial state of the game world very well and it soft decouples the inductive reasoning loop it relies on to know if it is on the right track.

The downside to this approach is it is far more labor intensive for CTF organizers, and requires players to have a computer capable of running the game. We are also betting on AI to not advance enough by the time we ship to be able to just ingest the entire game state in realtime and close the loop that way.

The cheating issue isn't really a matter of being able to run custom kernel code. You can do the same thing on Windows, which is why remote attestation is a thing for some games. As someone who has developed games for Linux (and Windows / Mac), it's an endless cat and mouse game. So long as the system can execute code that is not yours, you never really are getting perfect anticheat. Ease of loading custom kernel code isn't really a hurdle to that.

I find that client and server based in combination is the robust approach. I once implemented anti-cheat in which the server lied about game state, which a regular client without cheats would act predictably on. Deviation from that behavior is a useful heuristic to build a suspicion score.

Like the digital economy post .com burst, I think AI will survive and grow far beyond its current market of chat bots and agents. The weakest will die, but the market will be better off for it in the long run.

The next big problem for AI is time horizons. Frontier AI has roughly doctorate level knowledge across many domains, but it needs to be able to stay on task well/long enough to apply it without a human hand holding it. People are going to have to get used to feeding the AI detailed and accurate plans just like humans, unless we can leverage an expanded form of leading questions like GPT-5 does before executing "deep research". Anthropic feels best positioned to do this on a technical level, but I feel OpenAI will beat them on the product level. I am confident that enough data can be amassed to push time horizons at least in coding, which itself will unlock more capability outside that domain.

I feel it's very different from Tesla, because while Tesla barely ever got closer to their promises the AI industry is at least making visible progress.