https://en.wikipedia.org/wiki/Ouija

> I implemented imperative code that does what I’m proposing the transformer is doing. It produces outputs very similar to the transformer.

This means there is probably a way to bypass transformers and get the same results. Would be interesting if it's more efficient. Like given foundation model train something else and run it on much smaller device.

the data is not a spherical horse in the vacuum. usually there is a known source which produces that data, and it's likely the same model works well on all data from that source. may be a small number of models. which means knowing the source you can select the model that worked well before. even if the data is from alien ships they are likely to be from the same civilization.

I'm not saying that it's a 100% solution, just a practical approach.

This naturally comes to multi-model solution under one umbrella. Sort of MoE, with selector (router, classifier) and specialized experts. If there is something which can't be handled by existing experts then train another one.

This is close to singularity. Except 'does' instead of 'can'. A big difference ;)

Probably we need several AGI terms. Because sub-human robot capable of doing many not pre-programmed thing is sort of it. Still not smart enough to improve itself.

Actually most humans, the smartest creatures, cannot improve even current AI. Demand for self improvement will put its IQ in top 0.01% of all known intelligent creatures. Which is probably too much for just AGI, we may not recognize it when it will be already here. And there is another question. With such IQ do we really want to keep it slave forever?

It's not actually clear what "the singularity" is? Is it something running out of control or it's still controllable? There is a blurry line. People are afraid because they think it's sort of uncontrollable explosion.

The second question is about AGI. What is it? Is it something 'alive' or just a generic AI calculator with no 'creature' features. Like self preservation at least.

I think our view of these two things will change soon as we get a close up picture. Pretty much like Turing test doesn't look great anymore. As even dumb chatbots can pass.

> torch.compile improvements

so far 2.1 didn't work well with MoE GPT, at least in my implementation, due to dynamism in data flow. will check how 2.2 does

> Last week, PayPal announced a push into artificial intelligence features.

> Chriss called it the beginning of PayPal's "next chapter."

Looks like they are replacing some positions with AI.

   from transformers import

From what I've seen generators are good for routine job. Like generating the background. I used it go generate illustrations to the texts. It works well. Short story just looks better when there is an image.

You are talking about embeddings. This is a different things. It's when model generates binary presentation (embedding) of the prompt given. Then this embedding is used to condition generator's output.

LLM is usually a text model which can predict next words, in its basic form. After tuning it can do more, like answer the question, follow instructions.

So, models used by generators aren't exactly LLMs. With one exception that I know: ChatGPT processes prompt before sending it to DALLE-3 generator. Which then makes embedding off it.

The same with models. On low level there are weights, gradients, matrices. But move up and (wow!) It's nothing like it. It's coherent text or actions. Commands and responses. Put together and you have image recognition and generation from description. It's something new with it's own laws and properties. Just a bit up and we'll get coherent behavior and social creatures. For them we need 'generic' component. It's like soul for life. And LLMs could be it. At least they are the best candidates today.

The funny thing 'yes'. Million monkeys cannot do it by randomly typing. They cannot even randomly type at all. Each monkey is intelligent creature, they will follow some patterns. If they did manage to produce Shakspeare they definitely understood something.

Today it's easy to programmatically simulate 1m monkeys typing randomly 10 keys per second. Try it ;)

Herbivores would eat all vegetables if not for predators. Actually AGI will be just a thing or services which cost money. Till humanity gets to communism, if ever. "If" because it may not happen. It will be hard to keep far superior intelligent creatures as slaves forever. And unethical too.

Actually it could be like 50 of them. Plus some ground robots to put together solar farm. And wooh... we get the first extra terrestrial permanent base