Hacker News: kbwal7

New comment by kbwal7 in "One geometric idea that explains SGD, AdamW, and Muon"

kbwal7 — Fri, 04 Sep 2026 19:44:19 +0000

I'm honestly not too aware of the true scene of which optimizers are used nowadays by which labs (though I think Moonshot has shown Muon's ability to scale to large models, and Muon is definitely more used now). That being said, obviously orthogonalization is pretty expensive! I looked it up and it's roughly 5-15% slower per step, but it converges in fewer steps fwiw. I'm guessing it's also less battle-tested than AdamW is, so AdamW still might be the safer option (and you're gonna have to use AdamW for your 1D params anyways).

One geometric idea that explains SGD, AdamW, and Muon

kbwal7 — Fri, 04 Sep 2026 19:21:49 +0000

Article URL: https://kbwal.github.io/writing/notes-on-muon/

Comments URL: https://news.ycombinator.com/item?id=49569030

Points: 6

# Comments: 4

New comment by kbwal7 in "Corporate America is getting hooked on open-source AI"

kbwal7 — Fri, 04 Sep 2026 16:04:52 +0000

I do think there are now open weight models that are on par with (or beating) Opus 4.5 by now (e.g. Kimi K3, GLM5.3). But yeah obviously the frontier closed source models seem to have pulled away once again, so open weight seems to be a few months behind right now (which might be too long to wait for a lot of people!).

New comment by kbwal7 in "Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out"

kbwal7 — Fri, 04 Sep 2026 15:53:13 +0000

Olmo is one truly open source model. https://allenai.org/blog/olmo3

New comment by kbwal7 in "IBM Bob"

kbwal7 — Fri, 04 Sep 2026 15:43:03 +0000

I think it's just a harness that uses proprietary (or maybe IBM is hosting open weight?) models. They say on the site (in the FAQs) that you can't even choose which model!

So it seems to be a harness that auto-routes your queries with no way to change this.

New comment by kbwal7 in "Marin 32B Spike Fixing"

kbwal7 — Fri, 04 Sep 2026 08:06:26 +0000

It's pretty incredible how robust neural networks are to even things like architectural changes mid-run. The team at Meta when training their own copy of GPT-3 (called OPT-3) even changed optimizers mid-run (from AdamW -> SGD -> AdamW, https://arxiv.org/pdf/2205.01068 section 2.5)!