https://github.com/mmastrac/diffgemma

I've got a number of tiled-load kernels, and a top-k attention kernel that you might find interesting.

What I also learned is that MLX/vLLM is probably within ~20% or so of the absolute max perf on Mac. I found some improvements over what they were doing, but we're at the point where it's challenging to optimize without per-stepping kernels.

I found a few improvements over stock DiffusionGemma along the way, like using top-k attention, which drastically improves perf on my mac without sacrificing any of the benchmarks I was able to throw at it.

FWIW some of the issues with Gemma being slow on Mac are specific choices they've made in the architecture that make it challenging to make use various optimizations that have popped up recently. I think a Kimi K3-style network hybrid with the diffusion bits of DiffusionGemma could have some serious sway.

I think that diffusion still has an edge locally, but with some architecture tweaks and CPU improvements it would actually be a winner (ie: training the network for smaller token batch sizes or flexibility in attention heads, a less expensive attention mechanism, and others).

Feel free to reach out.

(currently at https://github.com/mmastrac/diffgemma but not in a releasable state yet)

FWIW I'm getting a hardware max of 20 tok/s (approx topping out the GPU's compute) on my custom local diffusiongemma port running on an M3.

I've been going smaller.. I have a custom-quantized Rust port of DiffusionGemma (26B) that seems to perform better (in responses) than benchmarks seemed to indicate and reasonably fast for its model size. Works really well on a 36GB mac as well for both prefill and generation.

It's been interesting learning about the balance of factors for performant metal kernels on unified memory.

Should have a repo up on github in the next few weeks.

The C2 IP (89.124.107.161) and malware-serving git repo (144.124.244.92) are both hosted on VDSINA in Russia, so not sure if there's anything to do there.

But I think the point is that they should be cheap to set up, and because of the short life, never really contain anything except the potential to compute when needed, not important data.

Comments URL: https://news.ycombinator.com/item?id=48687139

Points: 5

# Comments: 0

Luck surface area. I really owe so much to Om. I really can't imagine where I would be without that chance.

IMO, Deno has always been more methodical, more focused (maybe too focused?) on standards. But now the Deno team is on the right track: using Claude extensively to improve the node.js compat which was absolutely herculean if not impossible before AI. [+]

On the other hand, Bun has always played a bit fast and loose, chasing metrics at the cost of stability. Access to abundant AI has sent that project off the rails.

Disclaimer: Former Deno engineer - I'm obviously going to have some biases. All IMO of course, but if you ask me I'd still bet on Deno in the long term, and I personally still use it for any .ts projects.

[+] There might be a dozen people in the world that know how sensitive and subtle the timing and ordering in the JS event loop is and how meticulous just this single part needs to be for major node.js projects not to completely crap themselves.

There's a decent amount of knowledge around pre-main work in Rust, but I think this is one of the first attempts to walk through mutable link sections, which open up a pretty wide world of optimization, IMO. Even without mutability, I figured there isn't nearly enough documentation on these approaches out there.

(1) https://docs.rs/scattered-collect/0.20.0/scattered_collect/m...

(2) https://docs.rs/scattered-collect/0.20.0/scattered_collect/s...

Comments URL: https://news.ycombinator.com/item?id=48493512

Points: 93

# Comments: 25