https://github.com/Verilean/hesper

Even includes an example of transformer inference (quantized 1.5 bit):

https://github.com/Verilean/hesper/blob/a688ce9848d6416b2e95...

There's such a massive performance differential vs. SIMD though that I learned to appreciate SIMD (via highway) as one sweet spot of low-dependency portability that sits between C loops and the messy world of GPUs + their fat tree of dependencies.

If anyone want to learn the basics - whip out your favorite LLM pair programmer and ask it to help you study the kernels in the ops/ library of gemma.cpp:

https://github.com/google/gemma.cpp/tree/main/ops

gpupuzzles.answer.ai

Using WebGPU, you write code in the browser and computation runs entirely locally on your GPU.

Comments URL: https://news.ycombinator.com/item?id=41532298

Points: 14

# Comments: 2

Under examples/, for pedagogical purposes + help contributors understand what happens with WebGPU under the hood, I actually included an example of invoking the same GELU kernel as in the hello world example without gpu.cpp. It looks like this and is ~ 400+ LoC and also will take several minutes to build Dawn:

https://github.com/AnswerDotAI/gpu.cpp/blob/main/examples/we...

A goal of gpu.cpp is to make the power of webgpu much less painful to integrate into a project without having to jump through as many hoops (+ also sets up the prebuilt shared library so builds are instantaneous and painless instead of reams of cmake hassles + 5-10 minutes of waiting for dawn to build):

https://github.com/AnswerDotAI/gpu.cpp/blob/main/examples/he...

The situation has gotten a lot better for both dawn and wgpu integration in C++ with:

https://github.com/eliemichel/WebGPU-distribution/

Getting a shared library build was a revelation though, credit to:

https://github.com/jspanchu/webgpu-dawn-binaries

because the FetchContent cache invalidations would still periodically lead to recompiling which gets quite annoying. When it's just a matter of linking you get few-second builds consistently. The cost is we'll have a bit of hardening around potential ABI bugs but it's ultimately worth it.

We'll work towards wgpu support. There's some sharp edges in the non-overlap w/ dawn which seem most pronounced with the async handling (which is pretty critical), but I don't think anything is a hard blocker.

It's early but my current since WGSL -> SPIRV is fairly shallow mapping you should be able to get close modulo extensions. Extensions can be important though, in particular I'm tracking this closely:

https://github.com/gpuweb/gpuweb/issues/4195

One subgoal of gpu.cpp is to be able to have a canvas to experiment and see how far we can push the limits.

Will look into this in the coming weeks (or if anyone is up for contributing let us know).

If anyone adds bindings let us know so we can link it in the readme.

gpu.cpp is one level up - it's implemented using the WebGPU API, not an implementation of the WebGPU API. In theory it should work with both wgpu and dawn but in practice you find there's enough differences it takes some conditional branching + testing to support both.

Having both wgpu and dawn support would be nice and I think we'll get there in the coming months but for faster early iteration I wanted to keep things simple for now. There's implementation + maintenance + testing overhead that you start to have to carry around so it isn't free.

It's still early days for pushing compute use cases to WebGPU (OctoML being super early notwithstanding). There's a small matmul in the examples directory but it only has the most basic tiling optimizations. One of my goals the next few weeks is porting the transformer block kernels from llm.c - I think that will flesh out the picture far better. If there's interest, happy to collaborate + could potentially do a writeup if there's enough interest.

There's always some tradeoffs that comes with portability, but part of my goal with gpu.cpp is to create a scaffold to experiment and see how far we can push portable GPU performance.

You're probably better prepared than you think. The funny thing is after working on making compute workflows work with graphics APIs like vulkan and webgpu, CUDA is so user friendly by comparison :)

Feel free to say hi or ping us if you run into issues in the discord channel https://discord.gg/Q9PWDckbnR

However I was planning to announce next week after I've had a chance to test with my Windows-using colleagues and this thread came early, so it's possible we'll run into some hiccups.

Meet us on discord here if anyone needs helps or just wants to say hello - https://discord.gg/Q9PWDckbnR

I've had hour long conversations explaining the project talking about how webgpu can be used natively, how rust and zig people are using webgpu as a main GPU APIs (with wgpu and mach) and at the end there's still clarification questions about differences from WebGL and WASM.

The phrase "native webgpu" might as well be a Stroop Effect prank in technology branding.

First, there's already a few teams taking a stab at the vulkan approach like kompute, so it's not like that's uncovered territory. At the same time I first looked into this the khronos/apple drama + complaints about moltenvk didn't seem encouraging but I'd be happy to hear if the situation is a lot better.

Second, even though it's not the initial focus, the possibility of browser targets is interesting.

Finally, there's not much in the fairly minimalist gpu.cpp design that couldn't be retargeted to a vulkan backend at some point in the future if it becomes clear that (eg w/ the right combination of vulkan-specific extensions) the performance differential is sufficient to justify the higher implementation complexity and the metal/vulkan tug of war issues are a thing of the past.

Ultimately there's much less happening with webgpu and the things that are happening tend to be in the ml inference infra rather than libraries. it seemed to be a point in the design space worth exploring.

Regarding Dawn - I've lived where your coming from. Some non-trivial amount of effort went into smoothing out the friction. First, if you look at the bottom of the repo README you'll see others have done a lot to make building easier - fetchcontent with Elie's repo worked on the first try, but w/ gpu.cpp users shouldn't even have to deal with that if they don't want to. The reason there's a small script that takes the few seconds to fetch a prebuilt shared library on the first build is so that you can avoid the dawn build by default. After that it should be almost instantaneous to link and compile cycles should be a second or two.

But as I mention elsewhere in these threads, if the Dawn team shipped prebuilt shared libraries themselves, that would be an even better solution (if anyone at Google is reading this)!