New comment by mochizou in "Dust: Pretraining Transformers Without Backpropagation"

mochizou — Tue, 06 Oct 2026 11:29:10 +0000

Maybe the practical path is to separate fast-changing memory from slow-changing weights. Most things an agent learns during use probably don't need to become parameters immediately.

New comment by mochizou in "I built non-autoregressive decision models with RL a year ago"

mochizou — Sun, 20 Sep 2026 14:37:15 +0000

I tried Jev a bit, and the zero-shot performance already felt good enough to be useful. If you find the right place for that tradeoff, I don’t really see why “you could fine-tune BERT” is much of a criticism.

Hacker News: mochizou

New comment by mochizou in "Dust: Pretraining Transformers Without Backpropagation"

New comment by mochizou in "I built non-autoregressive decision models with RL a year ago"