(1) "How to take Smart Notes" by Soenke Ahrens

Recording the ticket with a video is everyone's first thought at defeating their restriction, and is no doubt the first thing they thought of when designing it. Hence, the codes expiring too quickly that you'll need a new video before you get through the line at the entrance of the venue. And messing with videos in a pressured line of people in front of a bouncer, is, as others have said, simply not practical for the vast majority of cases.

So it's kind of irrelevant - practically speaking - that it is possible.

While currently this is a relatively weak strength of genAI - assuming technological improvement of this technique over time, isn't it just as possible that the data quality will converge positively rather than negatively over time, in the future? That is to say, the web would be consistently "refined" as time goes on, by predominant VLLM?

Assuming that the internet is even "filled" as you say in the first place (personally I don't think organically-generated content is ever going to be pushed out of the internet, but that's my opinion, and I'll entertain the opposite case for the sake of the discussion). It also assumes that people are using models trained on the current state of internet "slurry" in the first place - that we are continually ingesting more internet YoY into these models. If we come up with a better model that needs less data to produce high-quality content, neither my nor your assertion is even relevant. Same case if the internet just decides to use small, low-quality models trained on only a portion of the internet.

But if the internet is continually recycling the entirety of itself through a model that has tens of millions of dollars of funding and research focused on directly improving the quality of it's answer metrics, it's not necessarily 100% locked into a downward quality convergence slide. Especially if we assert that humans /will/ continue to be consistently putting more organic data into the internet over time. It's a pessimistic take.

Are we trying to test if the candidate can solve the logic problem, or are we trying to test how they handle an /intentionally-confusing/ situation and what (psychological) biases they jump to with their solution?

If it's a test of their logic capabilities, then it seems like the numbers are artificially low, so maybe not so embarrassing as you say... Reason being, I think there are several confounding variables included in the results they'd need to control for if that was the point.

An obvious one, if we were testing logic directly, then I wonder if they allowed the participants submission to "show their work" rather than just which final cards they chose. Doing so would eliminate the "carelessness" confounder in the result where they didn't thoroughly think through all of the logical cases of the cards or where they accidentally included an incorrect card but understood the nature of the required solution, ie. if they knew they needed to disprove rather than confirm but accidentally included a useless card for disproving, they still understood how to solve the problem and thus the logic. What percentage of their results fall into that bucket?

There's also other confounding factors that are set up to "confuse" the participant here that could be removed if we wanted to truly test their /logical skills/ and /not/ some psychological/sociological property. For example, the question merely says: "test that if a card shows an even number". In English, "if" can mean both the inclusive or exclusive OR depending on context - it's needlessly vague, and additionally, I posit that in English, given the common usage of the phrase "test ... if", the phrase is /leading/ the participant to look for /positive confirmation of the rule/ rather than the negative. You can of course derive that the negative test is needed by studying the cards but why try to mislead them outright? Why not say "choose the set of cards that you'd need to flip to prove the rule is false"? This clearly demonstrates the task and doesn't send them on a goose chase.

There's other things too. It doesn't mention if these cards are from a global set of cards or the rule is only meant to be proven on the 4 cards presented. It implies the latter but if you start thinking about "confirming if the rule is true for all cards", it sends you down another useless logical rabbithole, yet, /cards normally come from a deck in real life/ and it is natural to expect there are more cards. Maybe if they wanted to be exact we shouldn't be using cards at all but rather wooden blocks or something.

And I'm sure there are more "biases" that I'm not catching here. If your goal is to test people's likelihood of affected by certain biases psychologically, then all's well and good with the test, go right ahead. But if you're going to present the poor results as some sort of indicator of an population's skill at logic, maybe not the best test without some better testing procedures, imo.

They're asking the wrong type of work from it. If you need some boilerplate or a transformation, it's going to give you a fantastic template to work with. If you need it, on the other hand, to engineer out a highly-specific and nuanced solution with an esoteric codebase to a complex problem, maybe not so much. The former is wide, the latter is narrow. It's going to take maybe a bit more breakdown of the scope into proper subproblems before you'll get a good answer; and that's something you can do yourself, or have an agent perform across multiple queries maybe (though I'll admit, more work needs to be done for the whole multi-agent workflows to be truly useful).