we can do much better with a cheap model on openrouter (glm 4.7, kimi, etc.) than anything that I can run on my lowly 3090 :)

I do not know much about the benchmarks but the two coding ones look similar.

ollama run deepseek-r1:14b

generally, if the model file size < your vram, it is gonna run well. this file is 9gb.

if you don't mind slower generation, you can run models that fit within your vram + ram, and ollama will handle that offloading of layers for you.

so the 32b should run on your system, but it is gonna be much slower as it will be using GPU + CPU.

prob of interest: https://simonwillison.net/2025/Jan/20/deepseek-r1/

-h

I am gonna check out emailengine for future work.

That said, I agree that I wish there were more done post-research towards products with some of this stuff.

If you're on linux or wsl2, I would run oobabooga with --verbose. Load a GGUF, start with a small number of GPU layers and creep up, keeping an eye on VRAM usage.

If you're on windows, you can try out LM Studio and fiddle with layers while you monitor VRAM usage, though windows may be doing some weird stuff sharing ram.

Would be curious to see the diffs. Specifically if there's a complexity tax in offloading that makes the CPU-alone faster but in my experience with a 3060 and a mobile 3080, offloading what I can makes a big diff.

we use openrouter but have had some inconsistency with speed. i hear fireworks is faster, swapping it out soon.

So yeah it's kinda smoke and mirrors. In some cases, for some long PDFs, it works really well. If it's a 500 page PDF with many disparate topics, it may do fine.

Are there good lemmy spaces for LLMs?

But the decisions they make are one of the things that can be automated. I do not know if you have been inside one of these places but the executives are not doing a great job deciding (at mine they decided opensearch was a better bet than elastic and switched existing installations).

A new regime came in and then bad decision after bad decision drove our best talent away. Consultants, everywhere.

Also, that number is much lower. Full time devs are down, contractors and consultants are up. As a full time dev at one of these places, it felt like the number of executives was growing as everything else shrank.

Perhaps you are right about the highest levels, but think about all of the middlemen executives and what they do.

And even that -- I think an AI could choose to not spend millions on Deloitte or Accenture on software that inevitably failed.

Not sure why those of us that live in code editors or even Excel should worry about our jobs more than those that live in Powerpoint.

And of course, as others have said, volunteering.

2. get a dog