I cannot afford the Cayenne, but once this gets to cheaper models this will be an option I would be willing to pay for.

Comments URL: https://news.ycombinator.com/item?id=49754212

Points: 1

# Comments: 1

Compared to OpenVINO (I tried ONNX runtime too, but never got it produce correct outputs with my quantized models) it is very nice to be able to build the exact kernels I need, at the quantization settings and precision that works for the models I have and with the custom operators required (speech models do a lot of non-standard stuff), run from a single set of sources, and not have to ship a hefty third-party DLL, and having to deal with their memory leaks and other stability issues.

As an example of you how can use it to deploy real models there is this project doing ASR and TTS: https://github.com/dropbox/nspeech .

Finally, I am also going to be switching the inferencing part of Witchcraft from current Candle on MacOS and OpenVINO on Windows to just Candle with Neso; https://github.com/dropbox/witchcraft

1) code gets split between a host part that goes through your normal compiler, and a device part that goes through the GPU compiler. You may as well write the kernels separate and compile them via a separate compilation step, and keep your trusted host compiler for the host-side code.

2) data needs to move between the host and devices via explicit buffer transfers and synchronization steps, CUDA tries to hide this with annotated pointers, but it is really easier to think about those as just buffers that you allocate and transfer IMO, instead of trying to transparently share pointers between host and device like CUDA does.

3) kernel launches can we wrapped in a function similar to:

void RunKernel(const char *kernel_name, size_t width, size_t height, size_t depth);

Instead of the funky <<< >>> syntax that CUDA for C/C++ imposes. The problem is that once you start putting that in your code, it stops being C++ and stops being portable to non-CUDA GPUs. The launching and grid settings can be a bit hard to grasp at first, but sugarcoating that in bastardized C++ syntax does not absolve from having to understand it eventually.

So a good place to start might be an OpenCL or Metal primer, depending on the hardware you have available. D3D12 (and probably Vulcan too) makes this much harder than it should be, with too much boilerplate but is overall a mature and well-designed API should you wish to develop for Windows. Starting with WebGPU might also be good these days. It has a very different shader language than the others, but the rest of the concepts are similar, and it has a strong emphasis on making things async, which is what you want for performance anyways.

Claude/Codex should be able to get you moving very quickly.