Hacker News: dogmaticdevhttps://news.ycombinator.com/user?id=dogmaticdevHacker News RSShttps://hnrss.org/hnrss v2.1.1Sun, 03 May 2026 08:30:39 +0000<![CDATA[New comment by dogmaticdev in "Show HN: 1gbps Tokenizer written in Assembly. 20x faster than HuggingFace"]]>I wrote this tokenizer using SSE2 SIMD Instructions. It takes text, removes white-space, and separates strings using a null terminator.

I didn't bother making it multi thread, since it is already very fast. Maybe I will one day.

stats: 10448 bytes in 11302 nano seconds 10448 ÷ 0.000011302 = 923620933.5 bytes, or 923mb/s

31346 bytes in 32241 nano seconds 31346 ÷ 0.000032241 = 972240315.1 bytes, or 972mb/s

As you can see, it approaches 1 byte per nano second as more text is parsed.

]]>Sat, 25 Apr 2026 16:29:04 +0000https://news.ycombinator.com/item?id=47902634dogmaticdevhttps://news.ycombinator.com/item?id=47902634https://news.ycombinator.com/item?id=47902634<![CDATA[Show HN: 1gbps Tokenizer written in Assembly. 20x faster than HuggingFace]]>Article URL: https://github.com/dogmaticdev/SIMD-Tokenizer

Comments URL: https://news.ycombinator.com/item?id=47902633

Points: 3

# Comments: 2

]]>Sat, 25 Apr 2026 16:29:04 +0000https://github.com/dogmaticdev/SIMD-Tokenizerdogmaticdevhttps://news.ycombinator.com/item?id=47902633https://news.ycombinator.com/item?id=47902633