Here is the chat template I used (and renamed to qwen.jinja): https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates/r...

For image support, drop to NGL 24 and add:

  --mmproj-url https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/main/mmproj-F16.gguf \

  --no-mmproj-offload \

  --image-min-tokens 1024 \

I guess since this is emulating the device instead of just passing the commands to vulkan it is easier to do DX11, but I could be wrong about this particular implementation.

I'm not sure the outcome would be beneficial for the US as a whole here. But perhaps that is not their priority.

I really just iterated over the harness over and over for about two weeks with opencode until I was sort of satisfied (still lots to do there :).

For the llama.cpp I asked claude fable to optimize it for my hardware and iterated a few times. In the end I landed on the following: https://pastebin.com/2PpJFUC0

At 10k context I get about 40 tps generation and 500 tps prefill. At 100k context I get about 25 tps generation and 400 tps prefill.

It works, but I often use gpt or claude to make a detailed enumerated plan of what I want to do first, then have qwen follow it.

I'm not sure if it is economical or not, but I have solar on the roof so the power use is not really an issue and I already have the hardware.

The biggest benefit for me is that it's all done locally, and I know the harness is not uploading anything or sending telemetry to someone else.

The quotes are in since the difference between the phases is 230V, but in practice it is almost the same as a 40Ax400V single phase connection.

I hope not though, perhaps I can pick up a H100 in a few years if they get sold on the open market.

I work in a bank and a can tell you that the customers absolutely hate ML when it rejects their loan application. Over the pond in the US, I have an impression that the fico score is not exactly popular either, but I have no first hand experience.

WoW has become quite soulless after the changes where everyone gets the same end game equipment but you have to grind upgrade currency to improve it.

My wife and I used to play quite a bit but it doesn't really engage anymore. Perhaps we are getting too old, but we should be in the correct customer segment (mid 40 years old).

Now I have spun up a local wotlk server with player bots powered by ollama which is actually a bit fun again.

[1] https://massgrave.dev/

Being able to sign contracts, engage with the healthcare system, file taxes, read messages from the government and do general banking without having to leave the home is a massive convenience boost.

We are a high trust society where the government or the banks are not out to "get you". The majority of the banks (not by volume but by numbers) are even in a structure without any ownership of the capital except for the depositors, and most of the profit from these banks that is not used to build the capital further is handed out to customers and/or the local community.

I know the Swedish government is also pressuring swish to integrate with vipps. So I guess you'll have this ability soon.

Due to this post I had to search a bit and it seems that llama.cpp recently got router support[1], so I need to have a look at this.

My main use for this is a discord bot where I have different models for different features like replying to messages with images/video or pure text, and non reply generation of sentiment and image descriptions. These all perform best with different models and it has been very convenient for the server to just swap in and out models on request.

[1] https://huggingface.co/blog/ggml-org/model-management-in-lla...

[1] https://www.nve.no/energi/energisystem/energibruk/stroemdekl...

[2] https://www.nrk.no/tromsogfinnmark/norges-siste-kullgruve-pa...

But 5 seconds / token is quite slow yeah. I guess this is for low ram machines? I'm pretty sure my 5950x with 128 gb ram can run this faster on the CPU with some layers / prefill on the 3060 gpu I have.

I also see that they claim the process is compute bound at 2 seconds/token, but that doesn't seem correct with a 3090?