I am in shock you wrote this.

More RAM means bigger models, which means smarter models.

Which is why Qwen and Gemma have been so interesting to a lot of us who run our own... Now 32gb VRAM isn't so bad, as these models can be run on that with decent results.

Where this gets interesting is in a couple years, when all the A100, etc, all the Enterprise hardware hits eBay.

Using Vulkan backend.

``` llama-server -fa on -t 7 -ngl 999 --mlock --fit off --kv-offload --no-webui --metrics --chat-template-kwargs {"preserve_thinking": true} -b 2048 -ub 1024 -m /mnt/models/unsloth/Qwen3.6-27B-MTP-GGUF/Qwen3.6-27B-UD-Q5_K_XL.gguf --mmproj /mnt/models/unsloth/Qwen3.6-27B-MTP-GGUF/mmproj-F16.gguf -c 262144 --kv-unified -ctk q8_0 -ctv q4_0 --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-ngl 99 --alias unsloth/Qwen3.6-27B-MTP-GGUF --temp 0.60 --top-k 20 --top-p 0.95 --min-p 0.00 --presence-penalty 0.00 --repeat-penalty 1.00 ```

LLamaCPP has had some massive updates in the last week or so.

All the inference happens on that card, so the CPU/RAM is there for the other containers.

I'll eventually swap the motherboard and CPU for something better, so I can fit 1 or 3 more of those cards.

Why not NVIDIA? 32gb on team green means spending crazy money. And I can get 4 R9700s for the cost of one 32gb 5090.

128gb ... Vs 32gb.

Basically I run a flow

Brainstorming > Create Spec > Review Spec* > Create Plans > Review Plan* > Execute Plan (in subagents) > Review Against Plan > Code Review* > Open PR > Finish Plan (marks plan files done)

* Each review step marked with an asterisk uses a paid larger LLM, right now Deepseek V4 Pro. Having it do this catches a lot of small things, and now I'm effectively one shotting any task I give it.

And it's not costing me much at all, just those three reviews. I could use a free model like Gemini but I'm happy with what I've got.

Boss is happy, very happy. We're rolling it out more widely now.

But this is the future.

Any variation of trackpoint where you have to move your hand away from the keyboard, is a failure IMHO

Which I'm happy for. So given that decision, I don't think it's unreasonable to think that they might be open to including Mmproj files in the GGUF.

Only issue I can think of is, which one? BF16, F16? Etc

Just totally forgetting that the frontier models themselves stole an insane amount to get to where they are.

It's theft all the way across the board, and when someone tries to make the argument that open models theft is bad, but Altman or Amodei's theft is good.. they are revealing a lot about themselves

Basically small and medium models that are crazy well trained for their sizes.

Then we have a lot of specular decoding stuff like MTP and others coming to speed up responses, and finally better quantisation to use less memory.

Local LLM is the future, and the larger labs know that the open models will eat their lunch once people realise that the gap is only a few months. If we were good with LLMs a couple months ago, we're good with the open models now.

That's exactly the kind of attitude that this discusses.

You create something that solves your problems, you put it up on GitHub, free, and open... Suddenly it turns out others have the same problems you did, your software solves them.

It starts ok. People are nice. But as it gains traction, a certain kind of toxic person becomes more and more common. The "YOU FIX IT NOW! I DONT KNOW" Kind of person.

You wake in the morning, look at your email, and it's a stream of being screamed at. That takes a toll.

All because you had an idea one time to build something that solved your problem you thought "hey I might just open source this".

> That's it. I don't need to hear about the struggle.

I get my DRAM needs at the RAM ranch.