<rss version="2.0" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Hacker News: yuhuabill</title><link>https://news.ycombinator.com/user?id=yuhuabill</link><description>Hacker News RSS</description><docs>https://hnrss.org/</docs><generator>hnrss v2.1.1</generator><lastBuildDate>Wed, 18 Nov 2026 19:30:57 +0000</lastBuildDate><atom:link href="https://hnrss.org/user?id=yuhuabill" rel="self" type="application/rss+xml"></atom:link><item><title><![CDATA[New comment by yuhuabill in "Show HN: Firenook – Firebase emulators in Rust: 40× faster writes, 10× less RAM"]]></title><description><![CDATA[
<p>Nice!</p>
]]></description><pubDate>Fri, 06 Nov 2026 15:46:43 +0000</pubDate><link>https://news.ycombinator.com/item?id=50393796</link><dc:creator>yuhuabill</dc:creator><comments>https://news.ycombinator.com/item?id=50393796</comments><guid isPermaLink="false">https://news.ycombinator.com/item?id=50393796</guid></item><item><title><![CDATA[New comment by yuhuabill in "JustFit: Run Qwen 27B Fast with its full 256K context on a 24 GB Mac"]]></title><description><![CDATA[
<p>Hi HN, I'm Yuhua. JustFit is a local inference service for Apple silicon that I build at night. v2 is built on Inco AI's Splash engine (their native runtime for Apple silicon) and their DFlash2 draft model. The goal: run a capable 27B language-and-vision model (Qwen3.8-27B, 4-bit) on a 24 GB Mac with its whole 256K context, fast enough to drive a coding agent.
The hard part is memory. The 4-bit weights alone take about 15 GB, so the KV cache, the speculative draft model, recurrent state, the vision tower and temporary buffers all compete for what is left. v2 keeps the KV cache in a 4-bit TurboQuant format that attention reads directly, loads components only for the phase that needs them, keeps a consistent committed state so conversations can be reused or restored from SSD after a restart, and splits work across CPU, GPU and Neural Engine depending on the chip.<p>Numbers from the paper, all on 24 GB-class Macs (M4 Pro, M5 Pro, M6):
- 256K in one request; 320K total across two concurrent requests with the draft model still resident
- 16K tasks: decode 2.0–2.5× my earlier MLX version (M5 Pro: 55 tok/s on one lane, ~110 on four); prefill 9–46% faster than Splash 1.3.0, the native engine this builds on (M5 Pro: 479 tok/s)
- AIME 2026, all 30 questions × 3 seeds, 4-bit weights and 4-bit KV: 94.4%
- a 128K conversation comes back after a restart in about 2 s instead of 445 s of prefill<p>For an end-to-end check I ran the same coding agent (dsh, the DeepSeek harness), same prompt and same reference image on one M5 Pro, once per engine: rebuild a Japanese garden picture as a live Three.js scene. JustFit v1 and Splash 1.3.0 were still going at the 4-hour cap (Splash compacted its 65K window 31 times); v2 finished in 1:06 at 163K tokens of context with no compaction. One recorded run per engine, so treat it as a demo; the two-minute replay is on the project page: <a href="https://yuhuachen.com/justfit" rel="nofollow">https://yuhuachen.com/justfit</a>
It is Apache-2.0, installs with pip (macOS 27, 24 GB+; tested on M4 Pro, M5 Pro and M6), and serves an OpenAI-compatible API. The paper (PDF) is in the repo; the arXiv version is pending.
Thanks to Inco AI for Splash and DFlash2, and to MLX and Prince Canuma's mlx-vlm, where v1 started.<p>Happy to go into the memory accounting on unified memory, speculative decoding with a resident draft, ANE vision, or what did not work.<p>Project page: <a href="https://www.yuhuachen.com/justfit" rel="nofollow">https://www.yuhuachen.com/justfit</a></p>
]]></description><pubDate>Thu, 05 Nov 2026 06:02:01 +0000</pubDate><link>https://news.ycombinator.com/item?id=50375743</link><dc:creator>yuhuabill</dc:creator><comments>https://news.ycombinator.com/item?id=50375743</comments><guid isPermaLink="false">https://news.ycombinator.com/item?id=50375743</guid></item><item><title><![CDATA[JustFit: Run Qwen 27B Fast with its full 256K context on a 24 GB Mac]]></title><description><![CDATA[
<p>Article URL: <a href="https://github.com/YuhuaBillChen/justfit-llm">https://github.com/YuhuaBillChen/justfit-llm</a></p>
<p>Comments URL: <a href="https://news.ycombinator.com/item?id=50375710">https://news.ycombinator.com/item?id=50375710</a></p>
<p>Points: 9</p>
<p># Comments: 1</p>
]]></description><pubDate>Thu, 05 Nov 2026 05:57:03 +0000</pubDate><link>https://github.com/YuhuaBillChen/justfit-llm</link><dc:creator>yuhuabill</dc:creator><comments>https://news.ycombinator.com/item?id=50375710</comments><guid isPermaLink="false">https://news.ycombinator.com/item?id=50375710</guid></item></channel></rss>