TFA is defending the use of AI, and it very clearly (to me) used AI to analyze the data and present the results.

In doing so, the author used statistics in a way they do not appear to understand, and ended up making numerous false claims (you can see the thread discussing these here https://news.ycombinator.com/item?id=48417626 )

In short, the study doesn't have sufficient statistical power, and is making "no difference" claims that aren't justified.

The meta-irony is this: the author used an LLM to interpret data in this study, and seems to have made the same category of mistake (confidently asserting falsehoods) that the study was supposed to be investigating (confidently submitting bad commits to the rsync project).

Of course the critical part is not the numbers, but what they mean.

So, what does the evidence mean?

The author interprets it to mean that there is no difference. They state this several times:

"46% EXACT PERMUTATION TEST P-VALUE (ONE-SIDED, H₁: CLAUDE MEAN > HISTORICAL)[...] What this p-value tells us is There's nothing unusual about the Claude group."

"74% ONE-SIDED P-VALUE (H₁: CLAUDE MORE LIKELY ABOVE MEDIAN) Fisher's exact test asks: if we split all releases at the historical median (0.74 sev/10c), are these Claude releases significantly buggy than previous releases (more likely to land above the median)? With a p-value of 74%, the answer is a decisive no. "

In an under-powered study, when a P value is above your alpha level cutoff (.05, .01, whatever was chosen) you can't distinguish between "no effect" and "could be an effect, but I didn't see one".

In an underpowered statistical study, a claim that two experimental conditions did not differ are not persuasive.

The ELI5 version is that there are two mistakes you can make when looking at a P value:

Type I error, where your P value is falsely low. In the experiment being discussed here, it would lead one to conclude that AI code is worse. Otherwise known as a false positive.

Type II error, where your P value is falsely high, leading you to conclude that AI code is no different. Otherwise known as a false negative.

https://en.wikipedia.org/wiki/Power_(statistics)

One can calculate statistical power for a given experimental protocol.

My hunch is that if you did this, you would find this experiment is grossly under-powered.

This means you can't make the "absence of evidence" claim.

ollama run $model "calculate fibonacci numbers in a one-line bash script" --verbose

  Model                         PromptEvalRate EvalRate
  ------------------------------------------------------
  qwen3.5:35b-a3b-q4_K_M         6.6            30.0
  qwen3.5:35b-a3b-nvfp4         13.2            66.5
  qwen3.5:35b-a3b-int4          59.4            84.4

But even a curl test on localhost was showing 1,2, or more seconds longer slowdown.s

After a lot of digging, realized that I was victim of some sort of DOS attack, which appeared to be a SYN flood attack.

In a normal SYN flood attack the SYN packets are sent from one IP address then never reply, leaving the server in a state with multiple connections stuck in the "SYN_RECVD" state.

In this attack, the SYN packets are actually being "sent" from multiple IPs - and one theory is that these are forged IPs, so the attack is really a reflection attack, where the attacker sends a forged IP SYN to my server, which replies (several times with SYN ACK)

I blocked the IP block /16 at the firewall and all was well, but this made me wonder:

How are modern OS's so vulnerable to this? On my macOS server, I could run netstat -anp tcp | grep '\.443 ' and see about 128 entries stuck in "SYN_RCVD" at which point the server just went to pieces.

In other words, if my server received 128 "SYN" packets, it would die for about 75 seconds.

Is this a macOS only problem? Are other OS's susceptible to this?

for this question, please consider "Use CloudFlare" off limits. I'm more interested in why modern OSs can be so fragile to a few (forged) TCP packets.

Comments URL: https://news.ycombinator.com/item?id=47293006

Points: 3

# Comments: 0

I just tried this (Ollama macOS 0.17.4, qwen3.5:35b-a3b-q4_K_M) on a M4 Pro, and it did fine:

[Thought for 50.0 seconds]

1. potato 2. potato [...] 100. potato

In other words, it did great.

I think 50 seconds of thinking beforehand was perhaps excessive?

[1] https://news.ycombinator.com/item?id=47202082

I just tried this (Ollama macOS 0.17.4, qwen3.5:35b-a3b-q4_K_M) on a M4 Pro, and it did fine:

[Thought for 50.0 seconds]

1. potato 2. potato [...] 100. potato

In other words, it did great.

I think 50 seconds of thinking beforehand was perhaps excessive?

Theory is that some of the model parameters aren't set properly and this encourages endless looping behavior when run under ollama:

https://github.com/ollama/ollama/issues?q=is%3Aissue%20state... (a bunch of them)

Curious, what does "He was 68 63" mean. Is it a bicycle gear joke about his age at death?

This is both right (Shift-X is the reverse of X due to convention) But is also wrong (Shift-Scroll is the macOS gesture for scrolling on maps where Scroll alone doesn't zoom in or out).

TLDR: I really wish Apple would adopt the "scroll up to zoom in" convention used by the rest of the free world.

But I see many references to it being called just "Aero", but some call it "Aero Glass" [1]

Does anyone know the truth?

[1] https://www.pcmag.com/archive/rip-aero-glass-windows-8-stick...

What is the name of the law when someone writes a think piece of "stuff I've learned" and fails to cite any of it to existing knowledge?

Makes me wonder if (A) they do know it's not their idea, but they are just cool with plagiarism or (B) they don't know it's not their idea.

Math is hard.

Imagine otherwise: a teacher who wants change their final exam from a 50 item Scantron using A-D choices, to a 50 item Scantron using A-E choices, because they think having 5 choices per item is better than 4, would need to ask for IRB approval. That's not feasible, and is not what happens in the real world of academia.

It is true that local IRBs may try to add additional rules, but the NU policy you quote talks about "studies". Most IRBs would disagree that "professor playing around with grading procedures and policies" constitutes a "study".

It would be presumed exempted.

Are you a teacher or a student? If you are a teacher, you have wide latitude that a student researcher does not.

Also, if you are a teacher, doing "research about your teaching style", that's exempted.

By contrast, if you are a student, or a teacher "doing research" that's probably not exempt and must go through IRB.

CFR 46.104 (Exempt Research):

46.104.d.1 "Research, conducted in established or commonly accepted educational settings, that specifically involves normal educational practices that are not likely to adversely impact students' opportunity to learn required educational content or the assessment of educators who provide instruction. This includes most research on regular and special education instructional strategies, and research on the effectiveness of or the comparison among instructional techniques, curricula, or classroom management methods."

https://www.ecfr.gov/current/title-45/subtitle-A/subchapter-...

So while this may have been a dick move by the instructors, it was probably legal.

"Next I tested llama.cpp running AI models over 2.5 gigabit Ethernet versus Thunderbolt 5"

Results from that graph showed only a ~10% benefit from TB5 vs. Ethernet.

Note: The M3 studios support 10Gbps ethernet, but that wasn't tested. Instead it was tested using 2.5Gbps ethernet.

If 2.5G ethernet was only 10% slower than TB, how would 10G Ethernet have fared?

Also, TB5 has to be wired so that every CPU is connected to every other over TB, limiting you to 4 macs.

By comparison, with Ethernet, you could use a hub & spoke configuration with a Ethernet switch, theoretically letting you use more than 4 CPUs.

For example, check out this mess:

> “Unfortunately, there is one significant issue with the aforementioned data: schooling. Seeing as the majority of work to date includes only aggregate data, it is impossible to account. The first concerns small N: seeing as most publish studies only include a handful of TRA data, there is a lot of room for error and over.

Unfortunately, there is a largely unaccounted for confound in this aggregate data which may make generalized analysis questionable: schooling.”