The transaction in question can remain opaque to the LLM and a %transaction.message% string is resolved in the layer between the LLM and the user.

I understand the point of distorted facts, but what I’m not sure how things are improved by basically having no trust in any facts?

But I have to say that the comparison is not really fair. Comparison is done with a 2 B model vs frontier models that are likely 100s of times larger. Also taalas with their 15000 tok/s inference are suspiciously missing from the comparison.

We need to see the comparison with this framework and useful models, which at present seems to mean ~30 B.

I wonder if it is captivating simply because it syncs cool graphics to audio, like those Winamp visualization filters in the old days.

Feedback: I think it would be beneficial to clarify that there is a trial at the top of the page. Currently we have download and buy buttons next to each other, which I assumed meant that the software can be downloaded first, but will require a key you get from ”Buy”.

The images show Switch 1 controllers. Since the Switch 2 is already out, does this imply that the tracking doesn’t work on Switch 2 controllers?

I don’t like the 3 PC activation thing. It implies that the software needs to call home and will stop working if the company goes under. Or, at least, will not be possible to install anew.

But I think the key is that in the standard autoregressive case we get memory bandwidth bound, so there are tons of idle compute resources. And so checking multiple tokens is cheap because we can batch and thus reuse the read weights for multiple tokens.

The verification step is similar to a prefill with a small batch size. The difference is what we do with the generated logits.

I personally would be wary about fire. Custom electronics without experience, and then putting (assuming here) high energy density batteries in a soft toy handled by little kids. Any accident will be absolutely disastrous. And when you scale up those very low probability failures are bound to happen.

How do you think about this, is this handled already?

Would be interesting to see a comparison to quantized versions of the other models. If this model beats the others also in a fair comparison it gives more credibility to it.

Trivially, with r=0, the error is 0, regardless of how heavily the direction is quantized. Larger r means larger absolute error in the reconstructed vector.

Bandwidth-wise it is worse (more bytes accessed) to generate and do random recall on than the vanilla approach, and significantly worse than a quantized approach. That’s because the reference needs to be accessed.

I guess implied is that since the KV cache is smaller, the probability is higher that the parts it that are needed are in fast memory, and that bandwidth requirements of slow links is reduced, and performance goes up.

Would be interesting with a discussion about benefits/drawbacks of the approach. Ideally backed by data.

> Instead of expecting it to understand my requests, I almost always build tooling first to give us a shared language to discuss the project.

This is probably the key. I’ve found this to be true in general. Building simple tools that the model can use help frame the problem in a very useful way.

One maybe obvious way would be that asking for rumors will actually creates the scenario that the NPC describes.

If, when the speaker actually stops speaking, there is a match vs predicted, the response can be played without any latency.

Seems like an awesome approach! One could imagine doing this prediction for the K most likely threads simultaneously, subject by computer power available, and prune/branch as some threads become inaccurate.