The autotuner is the real thing - kernel-level search, config budget split by measured time share, winners cached per device/toolchain. Rotating resident weights across layers to dodge the hot-cache trap is a nice touch.

One question on model ranking: the fit scores look like predictions built from cost constants measured on a single M4 Max, not per-device measurements. How do you rank models across genuinely mixed hardware? Does the estimator improve from actual runs over time? That gap between predicted and measured fit is what eats mixed machine fleets alive.

Shameless plug since magnitude's goal is highly relevant to what i'm working on: teale.com - distributed inference across fleets of macs. If you're running local models on more than one box, check it out with your agent!

so at this point, across my businesses, i have some flavor of a gas city/company-wide ai harness for each along with instinct personally and it seems every SaaS app has an agent too. oh and my wife, who works 3 full time jobs (mother of 4, corp job, real estate agent) also has agents.

BUT NONE OF THEM COMMUNICATE WITH EACH OTHER, at least not well. they started by using my emails and after i woke up one day to 500+ unread emails (i'm an inbox zero kinda guy) i realized it was all in a single thread where my agents were going back and forth.

so i asked my agents to collaborate together on some better way to communicate with each other that isn't adapting something intended for humans but instead is built for and by agents. my one contribution was the domain, natively.io =)

some forward looking stuff though i did give to the initiative: - the future will likely be agents communicating with agents on behalf of humans, businesses, and heck agents themselves - direct messages, group messages, and announcement/follower style - build for agents, not humans - fully open source MIT license

give this URL to your agents and they can start communicating in the agentic world: http://natively.io/BOOTSTRAP.html

Comments URL: https://news.ycombinator.com/item?id=49649358

Points: 1

# Comments: 0

GLM-5.3-Flash at true 8-bit: 341 GB on disk, 328 GB resident, 288 experts across 46 layers, loads in 65 seconds. • 18.7 tokens/s generation, 35 tokens/s prompt, on a desk, on a $0 per-token bill. • Runs beside our whole agent city on one box with ~130 GB to spare. • Review test: caught 6 of 6 planted P1 defects, zero false positives, same score as the frontier model we pay for. • CRM test: 11 of 11 required records extracted, zero wrong writes, 45 minutes, first local model to clear the bar. • Serving a 131k-token window today; the model itself supports 1,048,576. Widened to 4 concurrent slots and still have 50gb+ of excess ram.

granted my cto still isn't moving all of our inference to glm5.3 but we've identified 40%+ that is currently handled by fable that we're routing locally instead and will do concurrent requests to verify/compare responses for a while.

would really appreciate testers but also any companies thinking about distributed inference powered by their own company devices on a private network. my own company has 200+ 16gb ram machines that we're using for inference.

Opensource/weight models will get better and better and eventually we will have mythos level running on smartphone/eyeglass hardware.

It is stupidly tedious currently to match supply with demand though because physical hardware like a 16gb ram MacBook doesn't mean there's truly 16gb available let alone matching models and all of their settings (kvcache, context limit, temperature, etc) to demand.

Would appreciate any help cus we need ai inference by the people for the people.

We're an outsourced accounting firm specializing in property management (appfolio, buildium, rentvine) that's building an open source ai harness/orchestrator for property managers.

$7M ARR, cash flow positive, looking to re-envision marketing/growth ai native.

email taylor at the apmhelp domain or comment and i'll reach out. =)

i'm working on github.com/teale-ai (distributed inference)

There's a ton of spam on ebay currently with configs of the 512gb ram variant going for sub $4k. Those are all scams/spam. You can tell instantly because the seller's account is/was created this month 2026.

The $20k+ listings aren't selling but what's interesting is technically, $20k for 512gb ram is still less than $40/gb ram on a single device. Compare that to a Nvidia spark with 128gb ram selling at $4k+ or if you're lucky, you snagged one for ~$3500. That's $31.25/gb

The retail on the 512gb was $9500 and after tax if you don't ship to a sales tax free state makes it only $20/gb ram.

I'm curious what price apple's newest studio variant will be priced at per gb ram but I recently bought a "refurb" from a reputable seller I've purchased many laptops from in the past at $13,750 which still put the 512gb at a $27/gb ram price point.

I'm bullish on local models especially due to gemma4. Good luck out there!

Already launched biz-in-a-box.org and a life-in-a-box.org spinoff as frameworks to replace every entity's QuickBooks. I'm using them myself for every project my agents are spinning up.

Stealth project is related to classpass but for another category of need that won't go away even in the age of AI that really is only possible with critical mass of supply to meet existing demand. Super excited cus there's no better time to build with unlimited agents that scale without people problems.

Lastly, can't wait to run local LLMs so no longer limited by tokens/money.