Hacker News: anemll

New comment by anemll in "Getting 50 GB/S Back from the Apple Neural Engine"

anemll — Sun, 13 Sep 2026 02:08:52 +0000

Not all systems affected M1 and M5MAX are OK

https://x.com/anemll/status/2098454204478366132?s=20

New comment by anemll in "TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS"

anemll — Wed, 01 Apr 2026 20:03:41 +0000

Check it out, you might be able to speed it up using this https://github.com/Anemll/anemll-flash-mlx https://x.com/anemll/status/2038684375425200360

New comment by anemll in "iPhone 17 Pro Demonstrated Running a 400B LLM"

anemll — Tue, 24 Mar 2026 03:40:33 +0000

17B includes 10 expert plus one shared. So actual size of the expert is much smaller

New comment by anemll in "iPhone 17 Pro Demonstrated Running a 400B LLM"

anemll — Mon, 23 Mar 2026 19:19:23 +0000

Check my repo, I had added some support for GUFF/untloth, Q3,Q5/Q8 https://github.com/Anemll/flash-moe/blob/iOS-App/docs/gguf-h...

New comment by anemll in "iPhone 17 Pro Demonstrated Running a 400B LLM"

anemll — Mon, 23 Mar 2026 18:48:55 +0000

Thanks for posting this, that's how I first found out about Dan's experiment! SSD speed doubled in the M5P/M generation, that makes it usable! I think one paper under the radar is "KV Prediction for Improved Time to First Token" https://arxiv.org/abs/2410.08391 which hopefully can help with prefill for Flash streaming.

New comment by anemll in "iPhone 17 Pro Demonstrated Running a 400B LLM"

anemll — Mon, 23 Mar 2026 18:40:43 +0000

SSD streaming to compute units is new. M4 max can do 15 t/s with its 15GB/s drives

New comment by anemll in "iPhone 17 Pro Demonstrated Running a 400B LLM"

anemll — Mon, 23 Mar 2026 18:39:39 +0000

Yes, SSD speed is critical though. The repo has macOS builds for CLI and Desktop. It's early stages though. M4 Max gets 10-15 TPS on 400B depending on quantization. Compute is an issue too; a lot of code is PoC level.

New comment by anemll in "iPhone 17 Pro Demonstrated Running a 400B LLM"

anemll — Mon, 23 Mar 2026 17:52:35 +0000

multiple NAND, and apple already used it in Mac Studio. Plus better cooling

New comment by anemll in "iPhone 17 Pro Demonstrated Running a 400B LLM"

anemll — Mon, 23 Mar 2026 17:51:47 +0000

both, tbh

New comment by anemll in "iPhone 17 Pro Demonstrated Running a 400B LLM"

anemll — Mon, 23 Mar 2026 17:49:47 +0000

Probably 2x speed for Mac Studio this year if they do double NAND ( or quad?)

iPhone 17 Pro Demonstrated Running a 400B LLM

anemll — Mon, 23 Mar 2026 14:30:10 +0000

https://xcancel.com/anemll/status/2035901335984611412

Comments URL: https://news.ycombinator.com/item?id=47490070

Points: 713

# Comments: 326

New comment by anemll in "macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt"

anemll — Sat, 13 Dec 2025 03:51:59 +0000

Tensor Parallel test with RDMA last week https://x.com/anemll/status/1996349871260107102

Note fast sync workaround

New comment by anemll in "RDMA over Thunderbolt 5 on Apple Silicon – 14µs latency"

anemll — Tue, 25 Nov 2025 17:24:03 +0000

In macOS 26.2 (Tahoe) beta, Apple introduced a low-latency Thunderbolt 5 RDMA driver, enabling up to 80 Gb/s bidirectional bandwidth for Mac clustering—ideal for distributed ML on Apple Silicon. It's optimized for low latency, delivering ~14 Gbps throughput at 4K MTU. My tests (M4 Pro to M3 Ultra): Stock ibv_uc_pingpong achieved ~14 µs round-trip for 4K packets (requires GID index setup). Custom C++ variant hit 6-13 µs/iter: https://x.com/anemll/status/1993192776897642942 Code and details: https://github.com/Anemll/mlx-rdma/blob/anemll-rdma/ibv_roun... https://github.com/Anemll/mlx-rdma/blob/anemll-rdma/ibv_roun... (includes steps to enable RDMA in macOS Recovery OS terminal) Theoretically, this accelerates pipeline parallelism (faster layer handoffs) and tensor parallelism (low-overhead sharding) on GPUs, with potential extensions to ANE for real-time AI workflows.

RDMA over Thunderbolt 5 on Apple Silicon – 14µs latency

anemll — Tue, 25 Nov 2025 17:24:03 +0000

Article URL: https://twitter.com/anemll/status/1993182652204187929

Comments URL: https://news.ycombinator.com/item?id=46048147

Points: 6

# Comments: 1

New comment by anemll in "Qwen 3 now supports ARM and MLX"

anemll — Sun, 14 Sep 2025 23:55:01 +0000

It’s also supported in Apple Neural Engine https://github.com/Anemll/Anemll

Anemll adds Qwen3 support for Apple neural engine

anemll — Thu, 19 Jun 2025 23:22:43 +0000

Article URL: https://twitter.com/anemll/status/1935779931822420447

Comments URL: https://news.ycombinator.com/item?id=44323416

Points: 4

# Comments: 0

New comment by anemll in "Run LLMs on Apple Neural Engine (ANE)"

anemll — Wed, 07 May 2025 16:15:40 +0000

We can ran 2000 or 4000 context with ANE

New comment by anemll in "Run LLMs on Apple Neural Engine (ANE)"

anemll — Wed, 07 May 2025 16:14:22 +0000

M4 max should work at 120GB for ANE and 500+ for GPU. So GPU will be 3-4 times faster for anything over 1-3B. ANE is likely as fast for prefill due to higher FLOPs

New comment by anemll in "Run LLMs on Apple Neural Engine (ANE)"

anemll — Sun, 04 May 2025 14:28:48 +0000

Right.I was thinking about it, you still need batch refill, however, Apple Core ML tools were failing for attention activations quantization. Long context, pre-fill is still compute bound.

New comment by anemll in "Run LLMs on Apple Neural Engine (ANE)"

anemll — Sun, 04 May 2025 14:22:11 +0000

Yes for GPU, however ANE only supports FP16 plus integers. M4/A17 added accelerated int8 that is twice faster than FP16