The metric upon which their 10x claim is based (bits-per-byte) is exactly the metric which is optimized during pre-training. Post-trained models are fine-tuned to optimize other metrics, which is known to be detrimental to performance on bits-per-byte evaluations. So bits-per-byte evaluations will always make a pre-trained model look favorable in comparison to a comparable model which has also undergone post-training.

Can someone confirm whether the models they are comparing against (DeepSeek V4, Kimi K2, and Nemotron 3 Ultra) have been post-trained?

But optimizing the global structure -- things like defining the boundaries between kernels, the lifetimes of allocations (or when/how to reuse buffers), and scheduling/parallelizing operations -- seems inextricably tied to the type of traditional program analysis used by compilers.

Even tiling/layout decisions, which you mentioned can be optimized by hand or by an LLM in the context of a single kernel, must be optimized in the broader context of the computation to minimize copies/relayouts between kernels.

I think part of the problem is that massive models make the compiler problem much less interesting, since global program optimization reduces to local kernel-level optimization.

There's quite a bit of confidential information in the doc about the company and how it's positioning itself going forward to compete with US labs. I'd imagine they're not happy at all with this being leaked and are withholding investment as a punitive measure.

Not to mention the other interpretation seems illogical -- why would you pause fundraising if your perception was that you lacked resources compared to your competitors?

Tangentially related, are there any wearable devices that allow for high resolution respiration monitoring? I'm imagining some measurement of lung expansion over time (probably at least 10 Hz) so that I can quantify the deepness/shallowness of my breaths as well as the phase of inhalation/exhalation cycles.

The point of this announcement is to draw attention to the fact that the currently hyped topic is what they have been working on since their inception. If anything, it gives off a Schmidhuber-esque 'actually it was me who invented that' vibe. But trying to retroactively claim credit for the hype is nowhere near the same as following the hype.

As for your impression that the company is more generally hype-chasing, I'm really not sure how you would come to that conclusion. At the time of their founding, chatbots were the hype on the product side and model scaling was the hype on the research side -- topics they have largely eschewed. They instead were founded with a focus on evolutionary and collective intelligence and have maintained a fairly cohesive research direction ever since.

The concept you're touching on is the idea that LLMs (and humans) are functions which are inscrutable. Their behavior cannot be distilled into a series of logical steps that you can fit in your head, there are no invariants which neatly decompose their complexity into a few interpretable states, and the input and output spaces are unstructured, ambiguous, underspecified, and essentially infinite. This makes them just about impossible to reason about or compose using the same strategies and analysis we apply to traditional programs.

[1] Optionally, they can take in a source of entropy to add nondeterminism, but this is not essential. If LLM providers all fixed their prng seeds to a static value, hardly anyone would notice. I can't imagine there are many workflows which feed an LLM the exact same prompt multiple times and rely on the output having some statistical distribution. In fact, even if you wanted this you may just end up getting a cached response.

My hunch is that Google initially tried to play dumb to avoid compliance, as to not reveal they do in fact retain customer data. They had a plausible excuse as well -- the owner had no subscription so they don't store the data -- and took a gamble that this explanation would suffice until the situation resolved itself. I suspect that authorities initially took Google's excuse at face value, since they parroted this explanation to the public as well. As pressure mounted on authorities to make some headway on the case, they likely formally exercised whatever legal mechanisms they have at their disposal to force Google's hand, and only then was the footage released.

* To be more precise, these are bindings for the libav* libraries that underlie ffmpeg

I think you're overestimating the degree to which this type of research is motivated by big-picture, top-down thinking. In reality, it's a bunch of empirically-driven, in-the-weeds experiments that guide a very local search in a intractably large search space. I can just about guarantee the process went something like this:

- The authors begin with an architecture similar to the current SOTA, which was a mix of recurrent layers and attention

- The authors realize that they can replace some of the recurrent layers with attention layers, and performance is equal or better. It's also way faster, so they try to replace as many recurrent layers as possible.

- They realize that if they remove all the recurrent layers, the model sucks. They're smart people and they quickly realize this is because the attention-only model is invariant to sequence order. They add positional encodings to compensate for this.

- They keep iterating on the architecture design, incorporating best-practices from the computer vision community such as normalization and residual connections, resulting in the now-famous Transformer block.

At no point is any stroke of genius required to get from the prior SOTA to the Transformer. It's the type of discovery that follows so naturally from an empirically-driven approach to research that it feels all but inevitable.

As a side note, a similar phenomenon occurred with the Adam optimizer, where the ratio of public/scientific attribution to novelty is disproportionately large (the Adam optimizer is very minor modification of the RMSProp + momentum optimization algorithm presented in the same Graves, 2013 paper mentioned above)