About the second paragraph, first of all, thank you for the suggestions. Can I ask you to elaborate a little on the reasons for your proposals? For instance, even if redundant in some cases, I thought to myself it couldn’t be a bad thing to check for null pointers (though I could improve the error handling itself).

I’d put it this way: though the idea of ref counting felt very natural at the beginning for my use case, at some point I realized there were probably better techniques to achieve the same goal. I found myself multiple times writing nonsense like:

sum = value_add(v1, v2) mul = value_mul(sum, v3) [...] value_release(sum) value_release(mul)

so that later I could release the sum. When you only have one intermediate value it’s still acceptable, but at 3-4 it starts getting cumbersome.

After asking for feedback, someone rightfully pointed out that the better and faster approach for an autograd engine is using an arena allocator. My reason for saying “rightfully” is that arenas are ideal when you have many “objects” that have the same lifespan, such as the values involved in the forward/backward pass. RC is better when you have a lot of “objects” with independent lifespans.

The basic idea is the same as micrograd: each number is a `Value` node in a computation graph, ops connect nodes, and the `backward` function topologically sorts the graph before applying the chain rule in reverse. The C-specific parts are memory management and two simple data structures I needed to implement backprop: sets and vectors.

The source code is about 1,350 lines, MIT licensed, and well documented. Dependencies are just the standard library and libm. In addition, the repo contains two examples to showcase how the engine works: a toy regression and an MNIST task.

What this is not: a framework to build and train neural networks in production. Being scalar-valued makes it slow, and it wasn't built for numerical robustness or large datasets. There's no commercial aim here; it's a learning project.

If you read through it, I'd like to hear thoughts, both on the ML engineering aspect and on anything that reads as un-idiomatic C.

Comments URL: https://news.ycombinator.com/item?id=48570364

Points: 57

# Comments: 19