That Newton and Leibniz came up with similar ideas in parallel, independently, around the same time (what are the odds?), supports that.

https://en.wikipedia.org/wiki/Leibniz%E2%80%93Newton_calculu...

That is, rather than just translate activation to text, then text to activation, that final activation could then be applied to the neural network, and it would be allowed to continue running from there.

If it kept running in a similar way, that would show that the predicted activation is close enough to the original one. Which would add some confidence here.

But a lot better would be to then do experiments with altered text. That is, if the text said "this is true" and it was changed to "this is false", and that intervention led to the final output implying it was false, that would be very interesting.

This seems obvious but I don't see it mentioned as a future direction there, so maybe there is an obvious reason it can't work.

Vendors might also be running A/B testing or who knows what, even when you ask for a temperature of 0.

But, if you run a fixed model with temperature 0 on your local CPU, it will be deterministic (unless there are bugs).

So it would be more accurate to test the apps rather than the APIs, unless the goal is to warn people that just open chatgpt and ask there.

Per the article, the average human uses over 5 tons per year (Americans: 18). Adding 0.00072 to 5 is not really noticeable.

(There is also the cost of inference, of course.)

More generally, the ability to use tools is a form of intelligence, just like when humans and crows do it. Being able to craft the right Python script and use the result is non-trivial.

At least in theory.

What the user sees is the total behavior of the entire system, not whether the system has internal divisions and separations.

> For the multiplication task, note that agents that make external calls to a calculator tool may have ZEH = ∞. While ZEH = ∞ does have meaning, in this paper we primarily evaluate the LLM itself without external tool calls

The models can count to infinity if you give them access to tools. The production models do this.

Not that the paper is wrong, it is still interesting to measure the core neural network of a model. But modern models use tools.

They say they measured that cost, and it was most of the runtime in the old version (though they don't give exact numbers). That cost does not exist at all in the new version, simply because of the language.

It looks like neither is the "real win". both the language and the algorithm made a big difference, as you can see in the first column in the last table - going to wasm was a big speedup, and improving the algorithm on top of that was another big speedup.

Most of the web also doesn't use the Video element, but it isn't 'a dud' either.

Video and wasm are critical for a small subset of the web. That subset includes YouTube and Netflix for Video, and Figma and Photoshop and Unity games for wasm.

Now, maybe there aren't many because of performance - maybe they haven't used wasm because it was too slow. But I would appreciate seeing data on that - an application that tried wasm and gave up after seeing the overhead, at the least. But I would also expect to see apps that use wasm even despite some DOM overhead, because of the speedup on non-DOM code - and I'd like to see data on how much DOM overhead they are currently suffering.

I am asking because I'm familiar with a lot of apps ported to wasm, and they don't do this. That may just be because I am seeing one particular slice of the ecosystem! So I am very curious to learn about other parts.

What are examples of such applications? Honest question - I'm curious to learn more about issues such applications have in production.

> But we really shouldn't be requiring everyone to become an expert to benefit from wasm.

If the toolchain does it for them, they don't need to be experts, no more than people need to be DWARF experts to debug native applications.

I agree tools could be a lot better here! But as I think you know, my position is that we can move faster and get better results on the tools side.

About this specific point, it is unclear how much of a defect memorization actually is - there are also reasons to see it as necessary for effective learning. This link explains it well:

https://infinitefaculty.substack.com/p/memorization-vs-gener...

Correct, but this is has been one of wasm's guiding principles since the start: move complexity from browsers to toolchains.

Wasm is simple to optimize in browsers, far simpler than JavaScript. It does require a lot more toolchain work! But that avoids browser exploits.

This is the reason we don't support the wasm text format in browsers, or wasm-ld, or wasm-opt. All those things would make toolchains easier to develop.

You are right that this sometimes causes duplicate effort among toolchains, each one needing to do the same thing, and that is annoying. But we could also share that effort, and we already do in things like LLVM, wasm-ld, wasm-opt, etc.

Maybe we could share the effort of making JS bindings as well. In fact there is a JS polyfill for the component model, which does exactly that.