running an untouched, vanilla 4-bit version (Q4_0) I baked myself today (benched it against Q4_K_M (16gb) and IQ3_M (12gb), Q4_0 (15gb) is king)...

this model--

1: over 60% faster than qwen 3.6 version of the same dense 27b model, same engine setup (don't ask how, im not sure either)

2: has better reasoning quality, less "loopy" with its thinking patterns.. most certainly the smartest model on my roster currently

3: has the longest task horizon ive ever experienced (locally or otherwise)...I sent it a bunch of compressed ideas for an app, it sent me back the largest python app ive ever seen in one single ai response pass (80kb text file)

thanks qwen!! hoping to see the full model range get released...

I started with "llama-server" and custom stuff around it, which is great for single model setups.. but for multi-model harness with quick switching, llama-cpp-python is peak

Comments URL: https://news.ycombinator.com/item?id=49174981

Points: 2

# Comments: 0

the harnesses themselves are just as important as the models...different harnesses give different responses with the same prompt, same model...

if you have the 20/mnth claude sub or codex, you really should be using that to build a good local harness for yourself... claude won't be 20$ forever

build the stack first! when you get that new comp with massive ram, youre already set, just run a larger model!

big cloud models are incredibly good at building and teaching about local ai!

have fun in the rabbit hole!

if you are memory constrained like me, check out my custom models https://huggingface.co/macwhisperer

you should be foaming at the mouth to use claude or codex to make a custom harness, just for your own personal use with local models...

https://convert.neocities.org

https://huggingface.co/macwhisperer/Gemma4-12B-SuperDense

should run perfect for 12-16gb with maybe 10-20k context

seems intelligent enough that I would recommend this as a daily driver for friends who just want a local ai that can do most things relatively quickly (getting 10 tps on my m2 air)

it only "exists" when you talk to it.. much like your reflection in the mirror is only there when you're in view.

models can never be self-improving because it can never have "self". it can only mirror the appearance of self.

what's actually happening is "symbiotic group improvement".

our brains are resonant.. for those of use who are brilliant, getting leverage with ai just means that our innovative ideas become louder and more physically real every day.

eventually everything worth building will be built for free and made readily available.. no more "profiteering"

its Jevons paradox "efficiency breakthrough -> effort reduces -> growth potential rises -> transformative gains happen"...

some of us are in the "transformative phase"..

others haven't seen the "breakthrough moment" yet, but they will soon.