Speaking in terms of wall clock, the expensive part of decode is fetching the weights from memory. Predicting and validating a bunch of tokens using the already fetched weights is insignificant in comparison. Even if you have a poor acceptance rate for predictions, you won't really see a slowdown vs not using MTP.

Police are actively using flock to identify people traveling over state borders to buy marijuana and search their cars.

What actually happened:

Police used flock to locate someone with an active warrant for their arrest related to domestic violence. They noticed a pattern of crossing state lines at certain times, and used that to nab him. It turns out it was to buy marijuana and his car reeked of it, so he was charged with that, too.

It turns out Anglo is a diamond company.

The problem is California forcing new construction to buy overpriced solar from builders that makes little sense compared to commercial solar installations + grid.

- The musl allocator is only slow with multi-threading.

- Almost all other allocators have trouble reclaiming memory when using multi-threading. This often results in multiples more RSS than single threaded or musl's allocator.

Agree with you on mimalloc. It can even be configured to be aggressive in memory reclaim at the cost of performance.

If you tell me the model and the number of parallel streams, I will do the math.

I guess it depends on if you would be using ToU otherwise.

It looks like about 50% of Californians use ToU plans, but the number is only 10% nation-wide.

Caching really has nothing to do with this. With RAM offload you can mostly benefit from:

1) Batching for prefill is a huge win, even with MoE, since the batch sizes can be so large.

2) Keeping non-expert weights in VRAM, so the percentage of weights used per token in VRAM is higher. This benefit reduces with larger models, though.

> You can likely recover compute intensity and throughput by batching requests together, which (in practice, depending on sparsity) will end up reusing some of the loaded experts with high probability;

With MoE it's low probability.

> MTP then becomes applicable even for a MoE model

With MTP it becomes _extremely_ low probability.

"They already charge me to park my car, why can't I leave it there for a year for the same price as 1 week?"