Hacker News: gitpusher42

New comment by gitpusher42 in "Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac"

gitpusher42 — Wed, 29 Jul 2026 22:51:14 +0000

It is Apple platform only implementation because of Metal (and Swift). Other platforms would require CUDA or Vulkan and a complete rework

New comment by gitpusher42 in "Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac"

gitpusher42 — Wed, 29 Jul 2026 22:45:34 +0000

hm. just open repo, copy commands into your terminal and you will get app installed (if you have swift toolchain installed)

after that download 14gb of weights and enjoy offline inference (and a bit of Gemma4 intelligence) for your everyday tasks

multi turn chat is coming!

New comment by gitpusher42 in "Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac"

gitpusher42 — Wed, 29 Jul 2026 22:33:31 +0000

Thank you! If you can use it for your tasks I would be happy!

New comment by gitpusher42 in "Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac"

gitpusher42 — Wed, 29 Jul 2026 22:16:15 +0000

uh, I don't think it is possible to compare them. DwarfStar4 is for high end macs and a lot of ram. this project is more targeted to low end devices and "general use" Gemma4 model

New comment by gitpusher42 in "Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac"

gitpusher42 — Wed, 29 Jul 2026 21:59:35 +0000

Yeah, I checked it. One expert is about a 3.36mb block. If a cache miss happens I read whole block with one pread.

And there is some reuse. ~41% selected again for the next token, ~57% within two. Each layer has its own experts, so no reuse between these layers.

New comment by gitpusher42 in "Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac"

gitpusher42 — Wed, 29 Jul 2026 21:33:53 +0000

There was an ai winter for very long time. The math for NNs was already here, but not enough compute/data

I saw a pretty cool project to run an llm on an esp32 device https://github.com/slvDev/esp32-ai

New comment by gitpusher42 in "Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac"

gitpusher42 — Wed, 29 Jul 2026 21:16:47 +0000

Thank you!

afaik ollama relies on llama.cpp and mmap. mmap loads pages on demand and doesn't use the same explicit cache or parallel reads like my engine. Most likely ollama/llama.cpp will be way slower in this case

New comment by gitpusher42 in "Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac"

gitpusher42 — Wed, 29 Jul 2026 20:42:32 +0000

not with this engine. Kimi is a very different model. You can try to check HN later, I believe someone will build engine for this model for edge devices

New comment by gitpusher42 in "Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac"

gitpusher42 — Wed, 29 Jul 2026 20:29:35 +0000

Thank you! Let me know how it goes and share your tok/s results

New comment by gitpusher42 in "Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac"

gitpusher42 — Wed, 29 Jul 2026 20:23:35 +0000

I tried it. madvise didn't make mmap better than pread. I also tested F_RDADVISE, it helped on short decodes, but somehow got worse on longer decodes. Not very clear why, most likely problem somewhere at APFS and it is closed source and not much docs for it

New comment by gitpusher42 in "Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac"

gitpusher42 — Wed, 29 Jul 2026 20:17:24 +0000

m5 device also uses the same approach. the same 16 cache slots. and experts are evicted from memory as needed. And activity monitor shows 2gb usage for m5 pro (24gb btw)

New comment by gitpusher42 in "Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac"

gitpusher42 — Wed, 29 Jul 2026 19:59:59 +0000

Uh, not really, unfortunately. Asahi uses Vulkan for gpu, but kernels for this project are metal.

New comment by gitpusher42 in "Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac"

gitpusher42 — Wed, 29 Jul 2026 19:54:47 +0000

Yeah, correct! You can set up this engine to get more expert cache slots (e.g 32 instead of 16) to get a better hit rate and better tok/s. it will be 3.5gb instead of 2gb.

New comment by gitpusher42 in "Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac"

gitpusher42 — Wed, 29 Jul 2026 19:46:01 +0000

I double checked m2 logs. Cache hit rate is about 59-69%. 250-320MB went through `pread` per generated token. It is 3gb/s during this i/o phase.

New comment by gitpusher42 in "Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac"

gitpusher42 — Wed, 29 Jul 2026 19:31:23 +0000

Thank you very much!

I think it will throttle quite soon, but I haven't tried runs longer than 30minutes with this engine.

However, there is no constant load on ssd or gpu. i/o and gpu work are alternating and there is a brief idle periods for each i/o and gpu during inference (because gpu waits for i/o and after that i/o waits for gpu)

New comment by gitpusher42 in "Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac"

gitpusher42 — Wed, 29 Jul 2026 19:22:47 +0000

For Mac I would start from MLX engine. For exact model choice it is better to check bench results, and select model based on your need. A lot of good feedback about Qwen3.6, but I haven't used it in my tasks

New comment by gitpusher42 in "Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac"

gitpusher42 — Wed, 29 Jul 2026 19:09:03 +0000

Thank you! Text is not the main part of this repo. The main part is the technology and the list of experiments (and some useful knowledge I got from this project, haha). I have always been bad at writing or editing text (in both my native language and English), but without text it is impossible to share this project online.

Text was the last and most difficult part for me. It is not perfect (and this project is not perfect as well), but I believe it does the job of communicating my ideas

New comment by gitpusher42 in "Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac"

gitpusher42 — Wed, 29 Jul 2026 18:50:32 +0000

I am not native, my English is far away from perfect. I am using LLMs for checking my texts or grammar. I always trying to edit it properly, but sometimes I missing parts like that because I don't really have this "language feeling" as natives. Apologies for this

New comment by gitpusher42 in "Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac"

gitpusher42 — Wed, 29 Jul 2026 18:45:06 +0000

I think it depends on usage pattern. You trade speed for lower memory usage. Maybe engine specialisation and faster SSDs is the future for local inference, who knows

New comment by gitpusher42 in "Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac"

gitpusher42 — Wed, 29 Jul 2026 18:37:14 +0000

Thanks! I wanted to add hugging face token field to speed up model download, but then I realised that people might not trust to give their tokens And yeah, local models are better for security, at least your conversation stay on the machine