This feels like an essential pattern of the universe or something…

In any case I’m truly grateful for this site as a whole, the good and the bad.

https://pmc.ncbi.nlm.nih.gov/articles/PMC5241507/#B1

TLDR: NAC is a derivative of an amino acid called cysteine, as such it is a precursor for one of the most important antioxidants in the body and it can modulate key metabolic pathways associated with good health across a variety of organs, notably for decades it has been a universally successful antidote for acetaminophen (Tylenol) overdose, it’s available over the counter but NAC is not naturally found in foods, eating cysteine-rich foods like chicken turkey yogurt etc is the next best bet.

How about when you turn 18 or whatever the government gives you a signed JWT that contains your DOB? Anyone who needs to verify your age can check that and simply validate the signature via a public key published by the government.

Simply grab a new JWT when you need it, to ensure privacy.

And sure, sprinkle in some laws that make it illegal to store or share JWTs for clearly fraudulent intents.

> the vast majority of kids don't easily have access to alcohol or cigarettes

This feels like it comes from an affluent perspective, where I grew up it was trivial to acquire these things and much worse, there will always be someone’s older brother etc who will do this for $20 because he’s got nothing to lose.

You can find evidence of this in the literature, but it’s absurdly understudied, because big pharma would rather sell you a subscription to life.

Fortunately there are many good people in the world, especially in the field of medicine, who want to help their patients unconditionally. So there are glimmers of hope, like some of the top cardiologists in the world going against status quo and treating patients with fasting regimes instead of surgery.

Notably GCP is rolling this out to their DICOM store API, so you get the space savings of JXL but can transcode on the fly for applications that need to be served JPEG.

Only know this because we have tens of PBs in their DICOM store and stand to save a substantial amount of $ on an absurdly large annual bill.

Native browser support is on our wishlist and our contacts indicate the chrome team will get there eventually.

> I strongly believe it is one of the best technologies for AI agents

Do you have any quantitative evidence to support this?

Sincere question. I feel it would add some much needed credibility in a space where many folks are abusing the hype wave and low key shilling their products with vibes instead of rigor.

As a fellow Tri Dao groupie and lucky duck who gets to build on Hopper/Blackwell clusters, I find it amazing how difficult it is becoming to write kernels that saturate GPU hardware.

When I squint, there appears to be a trend emerging across work like FA4, monolithic (mega) kernels, etc. Namely, a subversion of the classic CUDA programming model in the form of fine grained task based parallelism, managed entirely in “user space”.

Not exactly sure what’s ahead but I’m strapping in for a wild ride…

This is buried deep in the PyTorch docs and I don’t have the willpower to go find it right now, sorry.

GPU kernels typically dominate in terms of wall clock time, the only exception might be very small models.

Thus the latency of tokenization can essentially be “hidden”, by having the CPU prepare the next batch while the GPU finishes the current batch.

https://news.ycombinator.com/item?id=44111673

I find it curious that fundamentals of the CUDA programming model (eg kernel launches) are being subverted in favor of fine grained task based parallelism that ends up using the hardware more effectively. Makes me wonder if CUDA has been holding us back in some ways.

What are the chances we see your work land in PyTorch as an experimental backend?

Awesome stuff thanks for sharing.

P.S. minor typo, your first two paragraphs under part 1 are nearly identical.

https://github.com/ScalingIntelligence/tokasaurus/blob/65efb...

I’m honestly impressed that a pure python implementation can beat out vLLM and SGLang. Granted they lean on FlashInfer, and of course torch.compile has gotten incredibly powerful in the last few years. Though dynamic shapes have still been a huge thorn in my side, I’ll need to look closer at how they pulled it off…

How close are we talking?

I’m not calling you a liar OP, but in general I wish people perpetuating such broad claims would be more rigorous.

Unsloth does amazing work, however as far as I’m aware even they themselves do not publish head to head evals with the original unquantized models.

I have sympathy here because very few people and companies can afford to run the original models, let alone engineer rigorous evals.

However I felt compelled to comment because my experience does not match. For relatively simple usage the differences are hard to notice, but they become much more apparent in high complexity and long context tasks.

Using DFloat11, tokens/sec was higher only when compared relative to running inference with some layers offloaded to CPU.

Classic comp sci tradeoff between space and speed, no free lunch, etc.

This is a common misconception of how MoE models work. To be clear, 17B parameters are activated for each token generated.

In practice you will almost certainly be pulling the full 109B parameters though the CPU/GPU cache hierarchy to generate non-trivial output, or at least a significant fraction of that.