Btw, thanks for the response. Not sure why you got downvoted for it, but you have my gratitude for being one of few devs who are sincerely responsive to these types of concerns.

Remember, Google aids and abets militaries of governments that the UN has found to be committing genocide.

Weird how people seem to forget this.

A highly extroverted person isn't going to make for a better overnight custodial worker than someone who prefers a more solitary lifestyle.

An actor who can tap into the emotional currents of high neuroticism in their work can offer a more sincere and authentic performance than an emotionally flat one.

Low conscientiousness correlates with risk taking and can be an asset in roles where over-planning to the detriment of acting can be costly - think firefighters.

E.g. if you are low in extraversion and agreeableness, you probably wouldn't make a good nurse or waiter, but you might not make a bad lawyer or engineer.

Ding ding ding, we have a winner!

Please do not ask me questions that I know nothing more about than AI. Wish there was something like LMGTFY but for AI.

Turns out, there is such a thing as a stupid question after all: any question that a chatbot can answer that winds up wasting the time of a real human being because the asker was too lazy or inconsiderate to use resources that don't waste anyone else's time first.

>If they wanted the generic LLM answer, they'd have gotten it in four seconds without involving you, which is, in fact, easier.

I hate to be the bearer of bad news but... while it can be seductively tempting to assume all humans act this logically, I must unfortunately be the one to inform you that, no, they do not, and no, they often don't get the answer that they were able to get themselves in four seconds without me, and instead choose to waste my time instead.

Some say it will never happen, but they said that about the now-dying tobacco industry, too.

>The best option is likely to rent hardware on Runpod.

Vast.ai is much cheaper, but the broader point here is contestable. The only dimension in which cloud GPU rentals win is cost. You lose the confidentiality, integrity, and availability benefits of local deployments.

Unfortunately, the prices rose on these a lot, but unevenly. Beelink GTR 9 Pro is $4400, Framework Desktop is ~$3500, for what is basically the exact same mainboard as a Bosgame M5 for $2800.

Apple's M5 Max is another attractive option. Apple silicon traditionally had great MBW and was good at TG, but struggled with PP, but the new neural engines in those GPU cores have made a big difference in a good way here.

Gorgon Halo is rumored for June announcement with Q4'26 release with basically +100 MHz clocks on Strix Halo, LPDDR5X-8533 instead of LPDDR5X-8000, but more importantly, 192 GB max instead of 128 GB.

I'd say it's better to wait for Gorgon Halo than to grab Strix Halo now. However, Medusa Halo, rumored for H2'27, is slated to have up to 26c Zen 6 (heterogeneous cores - kinds funny that AMD is heading towards these as Intel retreats from them), 48 CU of RDNA 5 instead of 40 CU RDNA 3.5, and a 384 bit bus w/ LPDDR6, which should make 256 GB at more like ~490-600 GB/s MBW, which will really make Strix and Gorgon Halo obsolete.

Also worth keeping an eye out for Serpent Lake (intel CPU + nvidia iGPU on a single board with unified memory, rumored for 2028-2029 iirc), and on the 160 GB Crescent Island Intel dGPU.

Running locally, you get confidentiality of knowing your tokens are only ever being processed by your own hardware. You get the integrity of knowing your model isn't being secretly or silently quantized differently behind the scenes, or having it's weights updated in ways you don't want. And you get the availability of never having to worry about an API outage, or even an internet outage, for local inference capacity.

And this isn't even starting to address the whole added world of features and tunability you get when you control the inference stack. Sampling parameters, caching mechanisms, interpretability etc.

OpenRouter may be cheaper than frontier labs, but you still lose all of these benefits from open weight models the moment you decide to rely on someone else's hardware for your processing.

M2.7 is smaller than DS4, 230B total params vs 284B total params. At any given quantization level, M2.7 will require ~19% less memory for the weights than DS4F at the same quantization level. Both can be quantized to arbitrary precision levels. Larger models like these quantize much better at lower precision than smaller models do. There is still loss, but it's less catastrophic in terms of usability degradation than for say, 27B or 14B or 8B models. Again, n=1, but M2.7 holds up phenomenally well for me with unsloth's IQ2_XXS UD.

>2. Just the KV cache of M2.7 would take ~50GB for 200k tokens AFAIK. It does not have the compressed KV cache that DS4F features.

KV cache weights can also be quantized. At Q8_0, this is essentially lossless. I can fit a 400k context window with Q8_0 KV cache quantization along with unsloth's IQ2_XXS UD weight quantization (plus my running OS) on a machine with just 128 GB of unified memory. Strix Halo, not Apple Silicon. There are more exotic approaches to KV cache quantization with much higher efficiency, like TurboQuant, but this is besides the point.

>3. The models are very similar in performances, despite all that. And DS4F is likely getting an update soon.

Yes, though it's worth noting that DS4F does require about 20% more total memory for weights at any given quantization level (284B vs 230B), will need to shuffle about 30% more data through the pipeline on every forward pass (A13B vs A10B), has much higher hallucination rates per AA, and hasn't been fully post-trained. DS4 isn't a base model, it has been instruct trained, tool trained, etc, but there is a lot of capability that has been left on the table as of current checkpoints, which are what's actually available now.

>So it is basically a quasi-frontier model that can run on a 96/128GB MacBook at large context windows. That's non trivial. Likely a coding version could be released in the future.

MiniMax M2.7 fits into this same box - quasi-frontier model that can run on 96/128GB unified memory platforms with a large context window. You're right that it's non-trivial. My preference comes in part from the fact that M2.7 already is coding focused, and had been out for almost 2 months before DS4F showed up.

By the way, in spite of my preference for M2.7 over DS4F (and for Vulkan over ROCm on my hardware), I'm a big fan of your work on DarkStar 4. I admire what you've achieved with the project, how much work you've put into it, and your willingness to share that with the world, too. Thank you for your contributions to the open LLM ecosystem.

I am currently serving Minimax M2.7 to myself at ~$0.015/1M blended tokens worth of electricity on my own local hardware, where I get all of the confidentiality, integrity, and availability benefits that are lost when choosing to run open weight models on someone else's API.

Open source means that all of the information necessary to recreate the final product is public, which in the context of LLMs, would include all of the training material, and build instructions (scripts to do the training). Very few models actually achieve this - Nemotron family is the only one that comes top of mind. A license to run, inspect, modify, and re-release is a good improvement on open weight models, but does not alone amount to the model actually being open source.

You are welcome to an alternative understanding of the definition of open source - as you correctly note, it's a contested term - just know that your definition is not the more widely accepted one that people think of when they hear "open source".

Your version of the term is much more aligned with the OSI, which was a federation of anti-FLOSS industry bodies created with the intent to capture, redefine, and weaken the original spirit of the FLOSS movement, which predates the OSI by almost a decade - the GPL was first released in '89, compared to the OSI's formation in '98 by members of the $10B for-profit Netscape Corporation, who's flasgship product was originally proprietary and was only open sourced after commercial failure against proprietary competitors.

None of this should be construed as an implication that I'm anti-open-weight. As I mentioned earlier, I think open weight models fulfill a lot of the spirit of open source. While a world where truly open source models are the norm is obviously preferable to a world where only open weight models are the norm, a world where only open weight models are the norm is still vastly preferable to a world where proprietary models running on other people's hardware is the norm.

I just think that we should be careful to avoid watering down terminology in ways that serve proprietary commercial interests over the interests of the public and of users. Open-washing is real, and it harms the intersts of users.