Maybe the way forward is in LCM or go JEPA, therwise, as this Apple paper suggests, we will just keep pushing the "pattern matching" further, maybe we get some sort of phase transition at some point or maybe we have to switch architecture, we will see. It could be that things change when we get physical multimodality and real world experience, I dont know.

That is what I am struggling with, it is really easy at the moment to slot LLM and make everything worse. Mainly because its output is coming from torch.multinomial with all kinds of speculative decoding and quantizations and etc.

But I am convinced it is possible, just not the way I am doing it right now, thats why I am spending most of my time studying.

I dream if err, if err dreams me.

I started writing forth few months ago, wrote few interpreters, with jit or with types etc, and its just amazing, I think anyone should do it. TBH I don't think any other exercise has thought me as much about programming as this.

I also notice the "return" of Forth, as it is probably the easiest high level language to make for computers with addressable memory and fetch execute cycle. The parser is just few lines of assembly and of course you can write the parser in the inner interpreter's bytecode, you don't even need assembly :) So hobbyists can just "make it" and make their own tiny operating systems with it. Of course everyone makes their own dialect, but I think thats OK. Things like https://github.com/howerj/lfsr LFSR CPU/VM running Forth, or UXN or duskos/collapseos.

Now you can also use language models to help you onboard into the language, it do some practice programs and rewrite one program in many ways.

So if you are young or old and never tried to Forth, don't miss out, its super fun.

I am sorry, by no means I think it is a trivial token predictor, or a stochastic parrot of some sort. I think it has a world model, and it can do theory of mind to us, but we can not do theory of mind to it. It has planning as visible from the biology of language models paper.

> So again, I disagree about AI-generated tokens having no meaning. But I would agree there is no human connection there

What I argue is that language is uniquely human, and it is how it is because of the human condition. I think we agree more than we disagree. I say that the meaning is 'halved', it is almost as if you are talking to yourself, but the thoughts are coming from the void. This is the sound of one hand slap maybe, a thought that is not your own but it is.

I guess I am saying is that AI is much more like Alien than Artificial, but we read the tokens as if they are deeply human, and it is really hard for people to not think of it as human, purely because it uses language in such profound way.

Popularity is somewhat proxy for 'good', but not always.

Is it going to be useful for language models to train on it? I think so, and I don't mind that. As long as they develop better world models and understand human language better.

The problem I have is with humans reading generated tokens. Human language is shared experience, the evaluation and interpretation of the symbols depend both on the author and the reader (even though many times they are the same entity).

When a person with HPPD says 'The sky is black', when the symbols enter your mind they are superimposed with your experience and their experience to create meaning. (HPPD is a disorder from damaged filters on the visual system, it seems that raw information from the eye sensors are entering the brain, and they can see the inside of their eyes when they look at the sky, so it looks black, as if the whole sky is filled with 'floaters)

When you read AI generated content, you are both the judge and executioner, the symbols mean whatever you want them to mean, they have no author (in the human sense).

So, I want to write for other humans to read :) Even if nobody reads it.

In gpt2 times I used to read gpt generated text a lot, I was working on a game to guess if the text is AI generated or not, and for weeks while I was working on it I had strange dreams. It went away when I stopped consuming tokens, in gpt4 age this does not happen as I am reading hundreds of times more tokens than back then, but I think it is just more subtle.

Now I use AI to generate thousands of lines of code per day, at minimum sometimes now I just blank out when the AI doesnt spit out the code fast enough, I dont know what am I supposed to write, which libraries it is using what is the goal of this whole function etc, as it is not my code, it is foreign and I honestly dont want to be reading it at all.

This week I took the whole week off work and am just coding without AI and in few days the "blank out" is gone. Well, I did use AI to read 300 page docs of st7796s and write barebones spi driver for example, but I treat it almost as an external library, I give it the docs and example driver and it just works, but it is somewhat external to my thought process.

People argue that all fields have evolved, e.g. there are no more blacksmiths, but I argue that the machinists now are much more sophisticated than the ones in the past, pneumatic hammers allow them to work better and faster, as they use the hammer they get better understanding the material they work with, as in the machine does not take away their experience and ability to learn. I always had 2 days per week where I code without any AI, but now I think I have to change the way I code with it.

AI is for sure making me worse, and lazy. And I am not talking about the "career" here, I am talking about my ability to think.

I wrote few days ago about it: https://punkx.org/jackdoe/misery.html

I am very happy that apple moved away from the quest of making the keyboard 0 height.

I have been making more and more time at night to reconnect with the craft.

I think more people in the community might be struggling, so I just wanted to share my way of dealing with it.

Please share how you are dealing with the tokens.

Comments URL: https://news.ycombinator.com/item?id=44060800

Points: 14

# Comments: 2

You could argue that those tools in the hands of skilled craftsman will create amazing things faster, but we all know what will happen is absolute flood of AI slop in every entertainment category.

I have been programming in all kinds of languages, from assembly to clojure, but in 25 years I never programmed stack languages, I was kind of scared of them, it wasn't until I read the book and made my own Forth I understood what I was missing. Since then I made few interpreters, with jit, or with types, etc, it was super fun, but most of all it allowed me to see a completely new paradigm of programming, kind of the first time you understand eval/apply from 13th page of the LISP 1.5 Programmer's Manual. A language that writes itself and it is written in itself.

If you are making your own Forth, this Brad Rodriguez's article is also really good [2].

[1]: https://archive.org/details/R.G.LoeligerThreadedInterpretive...

[2]: https://www.bradrodriguez.com/papers/moving1.htm

Comments URL: https://news.ycombinator.com/item?id=43972239

Points: 1

# Comments: 0

https://en.wikipedia.org/wiki/Rule_110

Comments URL: https://news.ycombinator.com/item?id=42509438

Points: 1

# Comments: 0

    | Name                                 | Semi-private eval | Public eval |
    |--------------------------------------|-------------------|-------------|
    | Jeremy Berman                        | 53.6%             | 58.5%       |
    | Akyürek et al.                       | 47.5%             | 62.8%       |
    | Ryan Greenblatt                      | 43%               | 42%         |
    | OpenAI o1-preview (pass@1)           | 18%               | 21%         |
    | Anthropic Claude 3.5 Sonnet (pass@1) | 14%               | 21%         |
    | OpenAI GPT-4o (pass@1)               | 5%                | 9%          |
    | Google Gemini 1.5 (pass@1)           | 4.5%              | 8%          |