We found that AI usage is basically guaranteed now, but certain challenge designs did thwart it. Challenges built with temporal visual elements made AI fall flat on its face, as it could not ingest/process the data fast enough to act on them in time. We also found that counterfactual challenges (ie. the result you get did not match what we suggested you'd get) made AI-assisted solve time slower compared to pure humans, indirectly penalizing over-reliance on AI. Multimodal challenges combining audio and visual elements were also very effective, but were not as accessible to players.

This paper gave us some ideas about designing those challenges: https://arxiv.org/pdf/2308.02950.

For our next event we figured out a way to thwart AI in our CTF: embed the CTF in a game engine. The loop essentially becomes something like this: Connect to a simulated access point in the game, the K8s cluster connects their attack container to a private network with the challenge box(es). Hacking the boxes doesn't render a flag, but rather changes in game state. AI did very poorly coping with this in our testing, as it can't derive the spatial state of the game world very well and it soft decouples the inductive reasoning loop it relies on to know if it is on the right track.

The downside to this approach is it is far more labor intensive for CTF organizers, and requires players to have a computer capable of running the game. We are also betting on AI to not advance enough by the time we ship to be able to just ingest the entire game state in realtime and close the loop that way.

The cheating issue isn't really a matter of being able to run custom kernel code. You can do the same thing on Windows, which is why remote attestation is a thing for some games. As someone who has developed games for Linux (and Windows / Mac), it's an endless cat and mouse game. So long as the system can execute code that is not yours, you never really are getting perfect anticheat. Ease of loading custom kernel code isn't really a hurdle to that.

I find that client and server based in combination is the robust approach. I once implemented anti-cheat in which the server lied about game state, which a regular client without cheats would act predictably on. Deviation from that behavior is a useful heuristic to build a suspicion score.