Hacker News: i_am_rocoe

New comment by i_am_rocoe in "Ask HN: What Are You Working On? (July 2026)"

i_am_rocoe — Mon, 13 Jul 2026 03:09:22 +0000

I'm working on a variation of MTP that recovers PP TPS (back to the same as with MTP disabled), keeping most of MTP's benefits to TG TPS.

https://news.ycombinator.com/item?id=48700782

Will propose a patch back to llama.cpp or provide it as a fork.

New comment by i_am_rocoe in "I patched llama.cpp to gain 20% prompt processing TPS. Help me make a PR"

i_am_rocoe — Sat, 27 Jun 2026 19:34:01 +0000

Thank you!

I patched llama.cpp to gain 20% prompt processing TPS. Help me make a PR

i_am_rocoe — Sat, 27 Jun 2026 19:03:20 +0000

I've been running Qwen3.6-35B-A3B locally on llama.cpp and noticed that prompt processing throughput gets too low with MTP. I got nerd-sniped.

What started as curiosity turned into a two-week rabbit hole of experiments and ended with a PoC that fully recovers the MTP PP overhead on GPU, above any expectation I had.

I'm not opening a PR to llama.cpp because this is AI-generated code, which goes against their contribution policy, which I support. If you know C++ and llama.cpp internals, I invite to work together with me to open a PR with a more mature implementation.

Comments URL: https://news.ycombinator.com/item?id=48700782

Points: 6

# Comments: 2

New comment by i_am_rocoe in "As; HN: I was curious why MTP affects PP TPS in llama.cpp. My PoC recovers it?"

i_am_rocoe — Thu, 25 Jun 2026 14:22:34 +0000

If you want to run the PoC locally to replicate:

Clone the masked-nextn-skip-catchup branch:

https://codeberg.org/rocoe/llama.cpp/src/branch/masked-nextn...

Run llama-server with at least --n-gpu-layers 99 --spec-type draft-mtp --spec-draft-n-max 2 --parallel 1 --no-cache-prompt.

I used Qwen3.6-35B-A3B-UD-Q2_K_XL MTP:

https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF?show...

I tested on a L40S (using Modal).

About the parameters:

--parallel 1, see Code review findings:

https://codeberg.org/rocoe/llama.cpp/src/branch/masked-nextn...

--no-cache-prompt, see Known limitations:

https://codeberg.org/rocoe/llama.cpp/src/branch/masked-nextn...

As; HN: I was curious why MTP affects PP TPS in llama.cpp. My PoC recovers it?

i_am_rocoe — Thu, 25 Jun 2026 14:21:09 +0000

I've been running Qwen3.6-35B-A3B locally on llama.cpp and noticed that prompt processing throughput gets too low with MTP. I got nerd-sniped.

I'm not a C++ dev, I know almost nothing about ML, and I'm only scratching the surface of how LLMs work. What started as curiosity turned into a two-week rabbit hole of experiments and ended with a PoC that fully recovers the MTP PP overhead on GPU, above any expectation I had.

TL;DR: instead of processing the last layer MoE FFN for the entire ubatch tokens (usually 512-2048 tokens), this PoC processes only the output row (usually 1 token during prefill). The result is PP TPS is back to the same as with MTP disabled, in my bench that was an uplift of 20%, keeping most of MTP's benefits to TG TPS, even with a slight drop in draft acceptance rate in one of the benchs. More details in the branch readme: https://codeberg.org/rocoe/llama.cpp/src/branch/masked-nextn-skip-catchup/README.md

I worked with GLM 5.1 to write the code, Minimax M3 ran the tests and benchmarks on Modal and GLM 5.2 reviewed the work. GLM 5.1 is very smart and GLM 5.2 is capable of spotting deep side-effects in the code, no surprise it's at the top. Minimax M2.x were fast but lazy, M3 is a real leap and deserves more attention: it is smart, proactive, follows instructions and auto-corrects.

I'm not opening a PR to llama.cpp because this is AI-generated code, which goes against their contribution policy, which I support. If you know llama.cpp internals, you're invited to take a look at the PoC. I'll be happy to work alongside you to open a PR with a more mature implementation. This work is released under MIT, same as llama.cpp.

Happy to answer questions in the comments.

Comments URL: https://news.ycombinator.com/item?id=48673852

Points: 4

# Comments: 1