I have plenty of real peers I interact with, I do not need that noise when I just need a quick answer to a technical question. LLMs are fantastic for this use case.

I get that even now it's very easy to let stuff get out of hand if you aren't paying close attention yourself to the actual code, so people assume that it's some fundamental limitation of all LLMs. But it's not, much like 6 fingered hands was just a temporary state, not anything deep or necessary that was enforced by the diffusion architecture.

Evolution is much less brutal and efficient. To you death matters a lot more than being trained to avoid a response does to ChatGPT, but from the point of view of the "tasty arsenic" behavior, it's the same.

The business reality is, you aren't Google. You can't collect a hundred million data points for each experiment that you run so that you can reliably pick out 0.1% effects. Most experiments will have a much shorter window than any analyst wants them to, and will have far too few users, with no option to let them run longer. You still have to make a damned decision, now, and move on to the next feature (which will also be tested in a heavily underpowered manner).

Posts like this say that you should be really, REALLY careful about this, and apply Bonferonni corrections and make sure you're not "peeking" (or if you do peek, apply corrections that are even more conservative), preregister, etc. All the math is fine, sure. But if you take this very seriously and are in the situation that most startups are in where the data is extremely thin and you need to move extremely fast, the end result is that you should reject almost every experiment (and if you're leaning on tests, every feature). That's the "correct" decision, academically, because most features lie in the sub 5% impact range on almost any metric you care about, and with a small number of users you'll never have enough power to pick out effects that small (typically you'd want maybe 100k, depending on the metric you're looking at, and YOU probably have a fraction of that many users).

But obviously the right move is not to just never change the product because you can't prove that the changes are good - that's effectively applying a very strong prior in favor of the control group, and that's problematic. Nor should you just roll out whatever crap your product people throw at the wall: while there is a slight bias in most experiments in favor of the variant, it's very slight, so your feature designers are probably building harmful stuff about half the time. You should apply some filter to make sure they're helping the product and not just doing a random walk through design space.

The best simple strategy in a real world where most effect sizes are small and you never have the option to gather more data really is to do the dumb thing: run experiments for as long as you can, pick whichever variant seems like it's winning, rinse and repeat.

Yes, you're going to be picking the wrong variant way more often than your analysts would prefer, but that's way better than never changing the product or holding out for the very few hugely impactful changes that you are properly powered for. On average, over the long run, blindly picking the bigger number will stack small changes, and while a lot of those will turn out to be negative, your testing will bias somewhat in favor of positive ones and add up over time. And this strategy will provably beat one that does Proper Statistics and demands 95% confidence or whatever equivalent Bayesian criteria you use, because it leaves room to accept the small improvements that make up the vast majority of feature space.

There's an equivalent and perhaps simpler way to justify this, which is to throw out the group labels: if we didn't know which one was the control and we had to pick which option was better, then quite obviously, regardless of how much data we have, we just pick the one that shows better results in the sample we have. Including if there's just a single user in each group! In an early product, this is TOTALLY REASONABLE, because your current product sucks, and you have no reason to think that the way it is should not be messed with. Late lifecycle products probably have some Chesterton's fence stuff going on, so maybe there's more of an argument to privilege the control, but those types of products should have enough users to run properly powered tests.

I guess this is a big and probably unbridgeable divide, some people think this sort of thing is obviously evil and others, like me, actually prefer it very strongly over a world where all advertising is untargeted but there is massively more of it because it's so much less valuable...

First: just like there can be individuals who lift up an entire team but are not ticking off tasks themselves, there can be apparently individually productive team members who slow the entire team down for any of a number of reasons. In my experience it's usually either that they are a) fast but terrible, b) have really weird coding styles that might not be bad but are very difficult for others to work with (architecture astronauts and people who barely speak the language often fall here), or c) are process bullies who try to set up the entire review system to enforce their preferences in a hardline way that suits them but delays everyone but them. Each needs to be dealt with very differently, to varying degrees of success, but my honest opinion at this stage is that no matter how productive these people seem by themselves it's mostly harmful to have them on a team. Behavioral issues in senior people tend to be really tough to adjust, and take a lot of energy from a manager that is better spent helping your top performers excel; that said, if you can get them to adjust sometimes it's worth the effort.

Second: pair programming works great for some people, but it is terrible for others. I've measured this on teams by trial and error over fairly long periods, and unfortunately it's also the case that people don't segment neatly based on their preferences, so the obvious "let them choose" method isn't ideal. There are pairs of people who really love pair programming and desperately want to do it all the time who are almost fully 2x as productive when split apart instead (yes, including downstream bugs and follow-ons, meaning that they really are just having two people do the job of one), and there are people who hate pairing who see similar multiples by being forced into it even though they hate it. My rough intuition is that there are two independent variables here, a social factor that measures how much you enjoy pairing, and a style factor that determines how much benefit you derive from it, with the two not correlating much at all. There might even be a slight anticorrelation, because the more social people who love it have already naturally done as much knowledge sharing as is helpful, and the people who hate it are underinvested there and could use some more focus on the fact that they're part of a team.

https://arxiv.org/abs/2402.09733

https://arxiv.org/abs/2305.18248

https://www.ox.ac.uk/news/2024-06-20-major-research-hallucin...

So I don't think it's that they have no concept of correctness, they do, but it's not strong enough. We're probably just not training them in ways that optimize for that over other desirable qualities, at least aggressively enough.

It's also clear to anyone who has used many different models over the years that the amount of hallucination goes down as the models get better, even without any special attention being (apparently) paid to that problem. GPT 3.5 was REALLY bad about this stuff, but 4o and o1 are at least mediocre. So it may be that it's just one of the tougher things for a model to figure out, even if it's possible with massive capacity and compute. But I'd say it's very clear that we're not in the world Gary Marcus wishes we were in, where there's some hard and fundamental limitation that keeps a transformer network from having the capability to be more truthful as a it gets better; rather, like all aspects, we just aren't as far along as we'd prefer.

It reminds me of the truth of the advice that my category theory professor gave us, that the definitions are both the least important and the most important things, simultaneously. They're the least important in that they're just words that wrap up very simple concepts, and merely knowing the definitions doesn't actually mean you can work with the concepts. But they're the most important thing in that most of higher level math really boils down to picking out the exact right set of definitions to use, at which point proofs tend to pop out as trivial and obvious statements using those definitions. And at a more practical level, you won't be able to read any math if the definitions are not ingrained, so you might as well get a head start and just rote memorize them if you want to succeed.

But it's interesting that the language is far less sticky in memory than the underlying intuition. My guess is that because the intuition is so much harder to develop, it wires itself in much more deeply than the words themselves, which can be pretty easily learned in a few hours of flashcard work.