opening this thread for questions/feedback if you have any

Either way, setting temperature to 5° is obviously wrong - even if it knew the current temperature - but models of this size can't reason about relative values very well.

Give it a tool to change temperature by a given amount, and see what happens!

could you say more? What kind of home assistant / what stack

yes you're right, there's only so much a 14MB model can do.

Needle excels at in-conext inference, with tightly defined environments. In our experience:

accurate descriptions + narrow tool scope = success

but if you're just looking for somewhere to try the model, use our in-browser playground! - https://cactuscompute.com/needle

In the meantime, if you have enough RAM for the current model (≈28MB), our repo will get you up & running:

https://github.com/cactus-compute/needle

We just shipped the biggest upgrade to our on-device inference platform:

- Built-in model confidence-based routing to hand off inference runs to the cloud - Converter for any PyTorch model - Lossless 4-bit quantization (evals on our GitHub README) - GPU acceleration on compatible devices (starting with Apple Metal) - Minimal RAM footprint - Runs on any Arm device: iOS, Android, Mac, DGX Spark, Raspberry Pi, and more

All in, a Gemma 4 E2B class model runs at 169 tok/sec on M5 Max, takes 2.7GB disk space with no accuracy degradation from FP16, uses 1.3GB of RAM, and requests help from cloud models when needed.

The problem we started with eighteen months ago: inference engines are built for datacenters, but consumer hardware has different physics: you share RAM with the OS, you get thermally throttled, and the same model behaves differently on different hardware.

So we wrote a runtime from scratch for resource-constrained devices. Since then, Cactus has grown to process millions of weekly inference runs, and tens of thousands of monthly active developers.

Our biggest learning from deploying Cactus in production apps is that while local models can handle 90% of workloads, that 10% gap means they're still not production-ready. Our users' fix was to build custom cloud fallback logic.

Cactus v2 fixes that:

Our approach to cloud fallback is to post-train a probe into the model's weights that reads its internal activations and emits a confidence signal. This way, the routing happens inside the model rather than in a prompt classifier sitting in front of it. We believe this is critical for multi-turn agentic work, where the model should know which turns are easy enough to be handled locally, and which are hard - and get handed off to the cloud. What ships today is single-turn routing for Gemma-4 E2B against a configurable escalation endpoint (Gemini, Claude, OpenAI-compatible, or your own endpoint).

Our next target is hybrid-native models for multi-turn agentic work. This is the genuinely unsolved problem. Our current probe-in-the-weights is showing promising results and we look to release the first model variants soon.

The hybrid variants are Gemma derivatives, released under the Gemma terms noted on the HF cards.

In addition to the hybrid routing, the runtime has SOTA quantization, which is lossless at 4bit, memory maps weights to decrease RAM footprint and runs cross-platform, with Python, Rust, React Native, Swift, and Kotlin bindings.

Disclaimer: Cactus is distributed as source-available - free for personal use and small companies; commercial license above that (Docker-style license).

You can get started on our GitHub: https://github.com/cactus-compute/cactus or by `brew install cactus-compute/cactus/cactus`.

Comments URL: https://news.ycombinator.com/item?id=48864459

Points: 1

# Comments: 0

We don't advise using GPUs on smartphones, since they're very energy-inefficient. Mobile GPU inference is actually the main driver behind the stereotype that "mobile inference drains your battery and heats up your phone".

Wrt to your last question – the short answer is yes, we'll have multimodal support. We currently support voice transcription and image understanding. We'll be expanding these capabilities to add more models, voice synthesis, and much more.

In our deployments, we've seen open source models rival and even outperform lower-tier cloud counterparts. Happy to share some benchmarks if you like.

Our pricing is on a per-monthly-active-device basis, regardless of utilization. For voice-agent workflows, you typically hit savings as soon as you process over ≈2min of daily inference.

https://cactuscompute.com/

Blazing-fast, cross-platform, and supports nearly all recent OS models.

re: "question would get stuck on the last phrase and keep repeating it without end." - that's a limitation of the model i'm afraid. Smaller models tend to do that sometimes.

Adding shortly!

to your previous point - Cactus fully supports tool calling (for models that have been instruction-trained accordingly, e.g. Qwen 1.7B)

for "turning your old phones into local llm servers", Cactus is likely not the best tool. We'd recommend something like actual Ollama or Exo