Hacker News: jephs

New comment by jephs in "Do transformers need three projections? Systematic study of QKV variants"

jephs — Fri, 05 Jun 2026 16:56:50 +0000

Scaling curves don't need to be drawn at particularly enormous parameter counts to be useful! If you can do a 300M and 1.2B run (like the authors do here), then you can do 150M, 300M, 600M, and 1.2B runs with only 50% more resources, and get a much better sense for whether effects seem to amplify or diminish as scale increases.

New comment by jephs in "Do transformers need three projections? Systematic study of QKV variants"

jephs — Fri, 05 Jun 2026 00:27:11 +0000

I'm terribly sorry, but scaling curves or GTFO. Any random pile of linear algebra works fine-ish at small scales. Very few random piles of linear algebra push the Pareto envelope at large scales.

New comment by jephs in "Retro-Tech Parenting"

jephs — Thu, 04 Jun 2026 18:43:04 +0000

I've got 5 & 6 year old kids. They have a a VHS player / tiny CRT monitor with a few dozen tapes, a tiny janky mp3 player with all my ripped post-y2k era albums, and lots of books and art supplies.

VHS tapes are so cheap. Every thrift store has hundreds for like half a buck each. All your friends have a box in their basement they want to get rid of.