Mojo's design goals would in theory help this problem long term, but my sm120 PTSD still lingers.

THe original MoE paper from Noam Shazeer et al. is worth a read on this bit, though the paper is admittedly pretty dense. But TL;DR is that each expert layer is learning highly abstract, localized structural and syntactic patterns in the data to minimize the loss function, and its doing this token-by-token (which in some cases may have some domain clustering, but that's just incidental).

When you start batching your queries, even if they all seem like theyre in a single domain, if you visualized the activations you'd notice that most if not all of the network is lighting up on the batched forward pass.

We have an internal eval that measures performance on tasks for a handful of embedded systems repos for our mmWave radios (mostly Rust, some C for microcontroller stuff). Qwen3.6-27B scores only 4% lower for pass@1, n=250 compared to Opus-4.8.

For the labeled dataset, the average PR size they're being measured against is around 1.5k SLOC.

This is very much "real-world usage" for us. The sort of change sets that come in daily/weekly and are solving non-trivial issues in the respective codebases.

As is usually the case, the most broad claims from both the labs and from the consequent pushback are talking past each other.

I have two GPU rigs both with 2x RTX Pro 6000, can get ~250 tk/s decode with deepseek-v4-flash in native mixed precision. For context, in antirez's dwarfstar project he only gets ~20-40 tk/s on the same model @ 2bpw on M5 Max.

The latter is for sure usable if it's your only option, but it's really hard for me to personally go back to speeds like that when I've experienced the former.

(Also worth noting dwarfstar only has experimental support for dspark spec dec, when that lands it will definitely give a big boost at higher acceptance rates)

CPUs were already becoming much more important during training for test-time scaling, but there you were still bottlenecked by GPU compute since the gradient updates back to the policy model are the actual gating factor.

During normal inference though, CPUs are becoming more of a bottleneck for more advanced workloads. Even if you have 20 agents running in parallel, if they're all compiling Rust concurrently your total wall-clock time per task is no longer bound by the decode throughput of the upstream model. You're just waiting for tools to execute. This gets compounded by VM/container overhead as well if you're doing the totally local sandbox approach.

It's definitely confusing from a presentation perspective, but they are somewhat coherent comparisons if you account for the inference heuristics involved.

(They could in theory be gaming the decode speeds with much larger than normal batch sizes given the TTFT is pretty high at around 8s)

Their work is based on the nemotron arch (so far).

And for MoEs, very small amounts of loss can mean you're flipped to entirely different experts (this is also a problem more broadly with numerical stability issues too).

Its also only 13B active, so your decode speed would be nearly 2x that of Qwen3.6-27B. So there are other latent benefits as well.

I'm empathetic to their position though. It is entirely unsurprising to me that someone working in a growth role at Dropbox is unimpressed by the current state of AI relative to its broader claims in the market. They're not working on AI itself nor are they using applied AI where you see the biggest gains (e.g. SWE, ML, data science, etc.).

We still have a significant capability overhang at the frontier for a big chunk of knowledge work task domains, so I think its understandable (given the above selection bias) why someone would think the confidence is overblown. They have a point in their own domain.

Also, at least a portion of this you could argue is arbitrary and entirely scoped to the eval itself. The fp8 GEMM score could be low simply because one of the shapes is fairly skinny (i.e. not enough math work to keep the compute engine busy for a meaningful amount of time).

The caveat though is that each new gen of hardware often comes with brand new constraints/features that a given generation of models haven't seen before (e.g. tcgen05 in blackwell was OOD at one point). As the models start to generalize better, this might not be a showstopper, but still an issue at least currently.

[1] https://kernelbench.com/

Ultimately I found the data model and UI to be both cumbersome and unintuitive. Langfuse ended up being the observability tool I went with instead over the one I built (and still use today).

"# TODO: implement a proper validator to compare against ground truth. For now we just check for exact string match on each line of stdout." [1]

This was one of my chief complaints about the entire R1 news cycle, it felt like no one actually read the technical report. They were being heralded for their openness, but they left out the most meaningful details that you'd need to reproduce their work.

[1] https://github.com/huggingface/open-r1/blob/1416fa0cf21595d2...

The newest biggest model can still matter even if you do not run every prompt through it. You'll always have some task where even small amounts of loss are unacceptable and thus you need to make sure frontier intelligence is used for it.

On the router point, yes, routing has some overhead. But the router does not need to run the biggest model to decide which model to use. We've been using tiny classifiers for recommendation engines for ages now, usually on CPU. If routing saves you from sending a large fraction of traffic to the expensive reasoning model, the routing overhead can easily be worth it.

> Also, if there is significant gains from caching, then like.. what are even doing here? Inputting something and then reading cached pieces of text based on their similarity to the input? Kinda like a search engine?

The caching I'm talking about is explicitly the attention/kv cache, so its not input similarity retrieval (that would be more like what you'd use in a RAG/IR system). Prompt caching is generally about reusing already-computed attention scores for repeated prompt prefixes. The idea being you don't recompute the same static system prompt, tool definitions, schemas, long shared context, or repeated boilerplate every time. In more sophisticated systems, you usually store multiple checkpoints so that a small prompt change doesn't result in all-or-nothing hit/miss scenario.

Sure you can. Just because there is a non-zero amount of margin pressure from the lower tier inference providers does not imply that revenue forecasts ought to be poor. Jevon's Paradox gets oversold in this current cycle, but I do think it's a relevant lens to view this through given how much demand has outpaced capacity.

The argument is that customers learning to spend less per task can be good for the viability of the market (really the total demand) even if it is bad for naive revenue-per-token assumptions. If a workflow goes from economically stupid to economically viable because you route 80% of it to cheaper models and reserve frontier models for the hard cases, that can expand total usage and improve cost per useful outcome.

But that is not the full argument he is making. If the claim is that the labs will not be able to pay their creditors because inference is structurally incapable of becoming profitable, then he absolutely needs to be right about the technical economics of inference.

One part of that is the balance-sheet argument (which already shows insanely good margins). But it also depends on how inference-time compute actually works: routing, batching, kv cache reuse, model segmentation, different latency tiers, etc. Much of those details he's just been straight up wrong about in his writing, so as a result I have to call into question the rest of his reasoning as well (in part to avoid Gell-Mann amnesia).