Comments URL: https://news.ycombinator.com/item?id=49928116

Points: 2

# Comments: 0

A few things make it harder: the panel contains questions drawn from multiple benchmarks rather than one recognizable test, the evaluation is automated and fixed ahead of time, and the raw outputs/results are public so odd behavior can be inspected.

But ultimately LiveNerf measures the behavior exposed through the API on a fixed public panel. It can’t prove what’s happening internally or guarantee the provider isn’t conditioning on the benchmark.

Longer term, I’d like to add held-out/private or periodically refreshed panels specifically to make benchmark recognition harder. I just don’t want to quietly change the current panel, because having a fixed instrument is important for the longitudinal comparison.

Also, it isn’t comparing one 10-day period once and calling it done. The window rolls forward daily, and a change has to clear the pre-registered 99% threshold in two consecutive windows before it’s flagged.

Ten days isn’t sacred, though. Once there’s enough longitudinal data, one of the things I want to evaluate is whether that window length is actually well calibrated or should be changed in a future version.

The actual 10-day windows contain substantially more samples than that validation, but I haven’t demonstrated that they’re sufficient to distinguish a same-family swap of that size, so I’m not claiming they are.

The goal isn’t to make the instrument say “nerfed.” A null result is a result too. I’d much rather publish “we couldn’t detect a change of this magnitude” than overclaim what the data can support.

The initial calibration screened 2,336 questions with 4 samples each, then selected the 78 questions where Opus 5.5 showed useful variance. The panel is run daily, and the actual decision is based on paired per-item differences across 10-day windows with clustered standard errors, not on any single day’s result.

The n=1 in the daily sampling rate means one sample per item per day, not one sample for the experiment. By the time a window is evaluated there are hundreds of observations, and a change has to clear a pre-registered 99% interval in two consecutive windows before LiveNerf calls it a change.

The nondeterminism is basically the reason the statistical part exists in the first place.

Comments URL: https://news.ycombinator.com/item?id=49896707

Points: 2

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=49871819

Points: 4

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=49833920

Points: 1

# Comments: 1

Comments URL: https://news.ycombinator.com/item?id=49809555

Points: 5

# Comments: 2

This is a fictional short story and no claim made reflects any real world event.

Comments URL: https://news.ycombinator.com/item?id=49756708

Points: 4

# Comments: 3

Comments URL: https://news.ycombinator.com/item?id=49666732

Points: 1

# Comments: 0

The sequence for Navier-Stokes was that these researcher spent a year working on it, then they published a possible breakthrough, OpenAI then spends $15M within a couple days to finish it.

This was incredibly opportunistic.

It seems like there’s a concrete error early on. The tail recurrence 1.4 is T_i + T_{i+1} = 1/(2i+1)² but in 2.12 it uses the factor (2X+3)². With the actual recurrence, K(i) does not vanish. So the zero count that forces deg K 4B+1 fails. At the extra zero at -3/2(2.16), which would win by exactly one no longer works.

The structure does not match how these results are proven. The paper mimics Calegari–Dimitrov–Tang’s 2024 proof for L(2,χ₋₃), but that proof was a deliberate capacity bound. Here the arithmetic content is replaced by a combinatorial minimization over index sets plus numerics no human has checked.

There’s also some minor sloppiness in where K=9 instead of K=0 in the definition of K and “nineteen century” typos which suggests not only was the math not proofread, but the grammar wasn’t either.

My guess is that an expert will probably find a specific gap quickly but it would be cool if I’m proven wrong.

Not that this matters much anymore as Google is now more an AI than it is a search engine, and an unreliable one at that.

Comments URL: https://news.ycombinator.com/item?id=49514874

Points: 4

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=49495725

Points: 3

# Comments: 0

When switching between google accounts on a random app or game, no issue. Quick code, you’re in. When I switch between accounts on Claude on mobile, I have to sign in, enter the code, screen freezes, I sign in again, can’t verify it’s me, I sign again, wait 30 seconds, I’m in.

I have to sign in three separate times when signing into a single google account in Claude. For every single other app it’s very quick.

I have a sneaking suspicion that Anthropic and google are working together to degrade sign-in performance for multi-account users. I can’t fully prove it yet but I plan to be able to in the week or two.