[1] https://www.physicalintelligence.company/blog/pi0

[2] https://en.wikipedia.org/wiki/Mode_collapse

If we’re subtracting one attention matrix from another, we’d end up with attention scores between -1 and 1, with a probability of effectively 0 for any single entry to exactly equal 0.

What’s more, the learnable parameter \lambda allows for negative values. This would allow the model to learn to actually add the attention scores, making a score of exactly 0 impossible.

Seeing the circuitry of a computer in this way helped me to understand that computers operated by means other than pure magic. And, the video I saw was much less descriptive of how a computer works than the one the OP linked. So, although neither video amounts to a full college course on the topic, there’s still a lot of value in their ability to expose people to the topic. It’s inspiring to see how computers are mostly a composition of NAND gates, and to compare the massive structures in the videos with the microprocessors of the real world.

[1] https://youtu.be/LGkkyKZVzug?si=hZRdmablPt15gGqn

[1] https://youtu.be/XfpMkf4rD6E?t=18m23s

1) An English dictionary as input.

2) List of words that start with "app" wiki page as input.

3) Other alphabetically sorted pieces of text.

4) Elementary school homeworks for spelling.

5) Papers on glyphs, diphthongs, and other phonetic concepts.

You begin to recognize that the tokens in these lists appear near each other in this strange context. You hardly ever see token 11346 ("apple") and token 99015 ("appli") this close to each other before. But you see it frequently enough that you decide to nudge these two tokens' embeddings closer to one another.

Your ability to predict the next token in a sequence has improved. You have no idea why these two tokens are close every ten millionth training example. Your word embeddings start to encode spelling information. Your word embeddings start to encode handwriting information. Your word embeddings start to encode phonic information. You've never seen or heard the actual word, "apple". But, after enough training, your embeddings contain enough information so that if you're asked, ["How do", "you", "spell", "apple"], you are confident as you proclaim ["a", "p", "p", "l", "e", "."] as the obvious answer.

Look at Ethereum v Ethereum classic. Same deal. We have two chains that share a common history, yet at some point the users of both decided to split and then society had to come to a consensus on what each chain would be called. Again, did the miners sit around and conspire to which chain would be called "Ethereum?" I don't think so. I think the decision was decentralized and emergent.

My point is, even if there was a nefarious actor who attempted a 51% attack, it seems like there would be enough of a societal pressure to ignore their empty blocks. There would exist a chain that would still be valued by the perpetrators, but not so much by the individuals being harmed by such an attack. The attacked chain would be maintained and acquire a new name "Bitcoin Hacked" or something similar, and the chain where society ignores the empty blocks would go on its merry way still being called "bitcoin."

[1] https://en.wikipedia.org/wiki/Three_Laws_of_Robotics

I still haven't made too much progress. It was easy to generate private keys. Then I had to implement ECC. That took me quite a while to nail. But now I've got public keys on the curve mentioned in this article. Now I'm still in the process of mapping this into the public key hash. I got the sha256 hash working pretty quickly using just the psuedocode on wikipedia. But public key hashes are ripemd160(sha256(publicKey)), and I'm still having troubles with the ripemd160 hash.

After that, I'll base58 encode the hash. Then there's a couple of BIPs I wanna implement, the HD wallets and the 13 word mnemonics ones in particular.

Basically, I think this stuff's important for me to know. All these math abstractions are chained together in a certain way that gives us cryptocurrency protocols, internet monies. In the process, I get a strong foundation in cryptography as well as domain knowledge of cryptocurrencies.

[1] https://github.com/bitcoinbook/bitcoinbook

[2] http://www.righto.com/2014/09/mining-bitcoin-with-pencil-and...

I've been tempted lately to model application state as the integral over time of all events (deltas) that have occurred. For example, imagine a simple game state:

  type GameState = {
    x: float; // x coordinate
    t: float; // current time
    v: float; // velocity variable
  }

  { x: 0,
    t: 0,
    v: 0, };

  [{t: 1, v:1}, {t:2, v:-1 /*this is deltaV, so back to v: 0*/}, {t: 3, v:2}, {t:5, v:-2}]

I've found this approach kinda sucks when you add things like collision detection. Your application would have to emit velocity deltas when objects would collide. If you've got a point that bounces around in a box with perfectly elastic collisions, then over time you'd have an infinite number of these collision/velocity delta events. But as you receive new events, all your precomputed collision events are for nothing (if your player logs out). So the traditional update loop simulate each tick as it occurs seems to work best.

Is there a more general way of thinking about this integration? Maybe with respect to another variable? Perhaps it would address this problem I'm having where I feel forced to quantize my game state onto ticks.

Wasn't too fond of my C# deep copy solution: var serialized = JsonSerializer.Serialize(this); return JsonSerializer.Deserialize(serialized);

I took an interested in functional programming, pure functions, immutability, etc. soon after.