Hacker News: marcobambini

New comment by marcobambini in "Run Kimi K3 using 29 GB of RAM at 0.50 tok/s"

marcobambini — Sat, 01 Aug 2026 10:02:33 +0000

Thanks guys for all the comments, I am going to rewrite the README (without using an LLM)

New comment by marcobambini in "Run Kimi K3 using 29 GB of RAM at 0.50 tok/s"

marcobambini — Sat, 01 Aug 2026 07:38:36 +0000

We are backed by the SQLite author, and we have the right to use the SQLite name. Please do your homework before writing such comments.

We used an Elastic license for some projects to protect our work from being used in SASS without a prior agreement. WASTE is and will always be available with a very permissive license.

I really don't understand why some people prefer to spread hate instead of just asking for clarification first.

New comment by marcobambini in "Run Kimi K3 using 29 GB of RAM at 0.50 tok/s"

marcobambini — Fri, 31 Jul 2026 18:10:38 +0000

I wrote tons of software, even a programming language by hand https://github.com/marcobambini/gravity.

I'm using my skills to orchestrate LLMs and agents, and I can write better code much faster. As developers, we can choose to adapt to new technologies or become extinct.

New comment by marcobambini in "Run Kimi K3 using 29 GB of RAM at 0.50 tok/s"

marcobambini — Fri, 31 Jul 2026 18:07:12 +0000

It is not yet available, the only way is to download the official Kimi K3 model and then convert it:

# 1. preflight: reachable? how big? does it fit? tools/fetch_weights.sh --dest /Volumes/staging/k3 --dry-run

# 2. download — resumable, safe to kill, safe to re-run tools/fetch_weights.sh --dest /Volumes/staging/k3

# 3. convert into a container uv run --with torch --with safetensors python tools/convert.py \ --src /Volumes/staging/k3 \ --out ~/models/k3.waste --jobs 3

New comment by marcobambini in "Run Kimi K3 using 29 GB of RAM at 0.50 tok/s"

marcobambini — Fri, 31 Jul 2026 18:05:54 +0000

We tried to use Metal, but for that specific project it was slower than just using NEON ARM optimizations. It is all documented in the docs.

Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

marcobambini — Fri, 31 Jul 2026 14:12:38 +0000

Article URL: https://github.com/sqliteai/waste

Comments URL: https://news.ycombinator.com/item?id=49123386

Points: 325

# Comments: 160

Running Kimi K3 on a local computer

marcobambini — Thu, 30 Jul 2026 16:52:58 +0000

Article URL: https://github.com/sqliteai/waste

Comments URL: https://news.ycombinator.com/item?id=49112587

Points: 2

# Comments: 0

New comment by marcobambini in "A new inference engine to run Kimi K3 2.78T parameter with 29GB of RAM"

marcobambini — Thu, 30 Jul 2026 14:54:18 +0000

Right, we trade speed for the ability to run a 2.7T-parameter model while preserving accuracy. It is a first version, and we plan to improve the inference performance.

A new inference engine to run Kimi K3 2.78T parameter with 29GB of RAM

marcobambini — Thu, 30 Jul 2026 14:01:44 +0000

Article URL: https://marcobambini.substack.com/p/the-waste-inference-engine

Comments URL: https://news.ycombinator.com/item?id=49110183

Points: 4

# Comments: 2

Show HN: Run Full Kimi K3 with 29 GB of RAM

marcobambini — Thu, 30 Jul 2026 06:12:55 +0000

Article URL: https://github.com/sqliteai/waste/

Comments URL: https://news.ycombinator.com/item?id=49106591

Points: 9

# Comments: 2

New comment by marcobambini in "Show HN: A new engine to run Kimi K3 on a laptop"

marcobambini — Wed, 29 Jul 2026 17:05:55 +0000

M5 Pro

Show HN: A new engine to run Kimi K3 on a laptop

marcobambini — Wed, 29 Jul 2026 15:41:50 +0000

Kimi K3 has 2.78 trillion parameters and ships as 1.42 TB of weights. It clearly does not fit in the memory of a laptop. But K3 is a Mixture-of-Experts model. For each token, only a small fraction of its 896 experts per layer is activated. That changes the problem: the entire model does not need to be resident in RAM, as long as the weights required by each token can be reached quickly enough.

We built WASTE — the Weight-Aware Streaming Tensor Engine — to explore that idea.

WASTE keeps the dense, repeatedly used part of the model resident in memory, stores the routed experts in an NVMe-optimized container, and streams only the experts selected during inference. The remaining RAM is used as a bounded expert cache.

The current Kimi K3 container is 982 GiB. On a 64 GB MacBook Pro, WASTE runs the complete model at around 0.32–0.34 tokens per second, with a measured minimum memory requirement of approximately 29 GB at a 4K context.

That is obviously not interactive performance yet. But the result we found interesting is that it works at all: this is the full open-weights model, not a distillation, a pruned version, or a smaller model using the Kimi name.

The engine is written in C and has no BLAS, CUDA, ONNX, or Python dependency in the inference path. The same code can be used through the CLI, embedded as a library, or exposed through the included OpenAI-compatible server.

Correctness was the first constraint. Every layer was validated against a PyTorch reference, with final logits matching within 3.6e-06. The vision tower is supported as well and matches its reference within 2.3e-06.

The current bottleneck is understood: K3 needs roughly 17 GB of expert data per token, and more than half of the decode time is spent reading experts from disk. The engine is already operating close to the measured throughput limit of the laptop’s internal SSD. The next improvements therefore need to reduce the number of bytes read per token and increase useful expert reuse without pushing the operating system into paging.

K3 is deliberately the extreme case. The same engine runs Kimi-Linear 48B from a 19 GB container at 8.92 tokens per second with an 8 GB memory budget. The broader goal is to make models that are much larger than available RAM usable locally, without sending private data to an API and without requiring specialized accelerator hardware.

We have published the engine, container format, conversion tools, benchmarks, validation suite, and also the experiments that failed rather than quietly removing them.

Everything is fully open source. Feedback on the storage layout, quantization, caching strategy, direct I/O, portability, and potential optimizations would be very welcome. Contributions of any kind — code, benchmarks, testing on different hardware, documentation, bug reports, or new ideas — are more than appreciated.

Repo: https://github.com/sqliteai/waste

Comments URL: https://news.ycombinator.com/item?id=49098966

Points: 7

# Comments: 3