Resources I’ve liked:

Sebastian Raschka book on building them from scratch

Deep Learning a Visual Approach

These videos / playlists:

https://youtube.com/playlist?list=PLoROMvodv4rOY23Y0BoGoBGgQ... https://youtube.com/playlist?list=PLoROMvodv4rOwvldxftJTmoR3... https://youtube.com/playlist?list=PL7m7hLIqA0hoIUPhC26ASCVs_... https://www.youtube.com/live/uIsej_SIIQU?si=RHBetDNa7JXKjziD

here’s a basic impl that i trained on tinystories to decent effect: https://gist.github.com/nikki93/f7eae83095f30374d7a3006fd5af... (i used claude code a lot to help with the above bc a new field for me) (i did this with C and mlx before but ultimately gave into the python lol)

but overall it boils down to:

- tokenize the text

- embed tokens (map each to a vector) with a simple NN

- apply positional info so each token also encodes where it is

- do the attention. this bit is key and also very interesting to me. there are three neural networks: Q, K, V – that are applied to each token. you then generate a new sequence of embeddings where each position has the Vs of all tokens added up weighted by the Q of that position dot’d with the K of the other position. the new embeddings are /added/ to the previous layer (adding like this is called ‘residual’)

- also do another NN pass without attention, again adding the output (residual) there’s actually multiple ‘heads’ each with a different Q, K, V – their outputs are added together before that second NN pass

there’s some normalization at each stage to keep the numbers reasonable and from blowing up

you repeat the attention + forward blocks many times, then the last embedding in the final layer output is what you can sample based on

i was surprised by how quickly this just starts to generate coherent grammar etc. having the training loop also do a generation step to show example output at each stage of training was helpful to see how the output qualitatively improves over time, and it’s kind of cool to “watch” it learn.

this doesn’t cover MoE, sparse vs dense attention and also the whole thing about RL on top of these (whether for human feedback or for doing “search with backtracking and sparse reward”) – i haven’t coded those up yet just kinda read about them…

now the thing is – this is a setup for it to learn some processes spread among the weights that do what it does – but what those processes are seems still very unknown. “mechanistic interpretability” is the space that’s meant to work on that, been looking into it lately.

The main point being: the implicitness is not necessary for "compiler makes sure you don't forget". So the original comment about how usage of the explicitly named and paired APIs can clarify intent both for the writer and reader can still stand while not implying that forgetting is involved. I see this dichotomy often being drawn and I think it's important to consider the language design space more granularly. I think a reminder is a better fix for forgetting, rather than assuming what you wanted and doing it for you.

(the explicit calls also let you / work better with "go to defintion" on the editor to see their code, learn what to look up in a manual, step in with a debugger, see reasonable function names in stack traces and profiles, pass / require more arguments to the drop call, let the drop call have return / error values you can do something about (consider `fclose`), let the drop call be async, ...)

You do have to tag struct fields with a macro, but you can attach contexpr-visitable attributes. There's also a static limit to how many reflectable fields you can have, all reflectable fields need to be at the front of the struct, and the struct needs to be an aggregate.

Re: the comments on here about the GC etc. -- I've posted about this a couple times before but I've been using a custom Go (subset / some changes) -> C++ compiler for hobby gamedev, which helps with perf, gives access to C/C++ APIs (I've been using Raylib and physics engines etc.) and also especially has good perf in WebAssembly. Another nice thing is you can add in some reflection / metaprogramming stuff for eg. serializing structs or inspector UI for game entity properties. I was briefly experimenting with generating GLSL from Go code too so you can write shaders in Go and pass data to them with shared structs etc.

The compiler: https://github.com/nikki93/gx (it uses the Go parser and semantic analysis phases from the standard library so all it has to do is output C++) (it's a subset / different from regular Go -- no coroutines, no GC; just supporting what I've needed in practice -- typechecks as regular Go so editor autocomplete works etc.)

A game made with it a while ago for raylib game jam along with the raylib bindings and other engine stuff: https://github.com/nikki93/raylib-5k

Very old video of a longer term game project I'm using it on which also shows state-preserving reload on rebuilds and also the editor inspector UI: https://youtu.be/8He97Sl9iy0?si=IJaO0wegyu-nzDRm (you can see it reload preserving state across null pointer crashes too...)

You do need some JS code that asks the browser to run the wasm blob. You can't eg. just have a script tag that refers to a wasm blob yet.

libc does help with things like having an allocator or string operations etc., or for using C libraries that use libc. And that's where emscripten becomes helpful.

Browser functionality like the console or making html elements is exposed through JS interfaces, and the wasm needs to call out to those. But they may be directly exposed to wasm later (or they may already be at this point in new / experimental browser features).

I'm trying to decide how much time I should devote to making this easier to set up / use by other people in the medium term, since it's just a side project for me. Might make a codespaces template so it's quick to get started.

I ended up writing a Go -> C++ compiler. Go's standard library parser/typechecker made it very doable. The games I've done with it don't use the GC at all but also don't manually manage memory -- they use an ECS api which helps. https://github.com/nikki93/gx -- the README links to development workflow video and complete example game code. I get the perf, interop/libs and portability of C/C++ but with Go's developer experience (well the build system involves C/C++ of course but I have something set up there that I now just use).

I think I've seen wasmtime before. If I needed to interface to any C/C++ things on the server I would probably just write in C/C++ (or Gx) yeah.

What makes you say Wasm is a server side runtime / imply that it's meant to be one?

But yeah Wasm has definitely been used successfully with performance results in production, eg. in Figma: https://www.figma.com/blog/webassembly-cut-figmas-load-time-...