Comments URL: https://news.ycombinator.com/item?id=47283038

Points: 2

# Comments: 1

Comments URL: https://news.ycombinator.com/item?id=42419072

Points: 2

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=42413765

Points: 52

# Comments: 23

A lot of tech roles are intellectually stimulating and have a wage in the upper distribution of salaries. I think we have a very nice situation going for us.

Continual/online learning is still an area of active research.

I do think it is an extremely inefficient way to have a swarm (e.g. across time through training data) and it would make more sense to solve the pretraining problem (to connect them to the external world as you pointed out) and actually have multiple LLMs in a swarm at the same time.

> If you pretrained an LLM with data saying Moscow is the capital of Connecticut it would think that is true.

> Well so would a human!

But humans aren't static weights, we update continuously, and we arrive at consensus via communication as we all experience different perspectives. You can fool an entire group through propaganda, but there are boundless historical examples of information making its way in through human communication to overcome said propaganda.

The situation you described is possible, but would require something like a subverting effort of propaganda by the state.

Inferring truth about a social event in a social situation, for example, requires a nuanced set of thought processes and attention mechanisms.

If we had a swarm of LLMs collecting a variety of data from a variety of disparate sources, where the swarm communicates for consensus, it would be very hard to convince them that Moscow is in Connecticut.

Unfortunately we are still stuck in monolithic training run land.

This is probably why.

I issued a chargeback.

- We don't have limitless CPU cycles

- Thus we need to split things into sub-problems

If so that might still be amenable to the bitter lesson, where Sutton is saying human heuristics will always lose out to computational methods at scale.

Meaning something like:

- We split up the thought to vision problem into N sub-problems based on some heuristic.

- We develop a method which works with our CPU cycle constraint (it isn't some probe -> CPU interface). Perhaps it uses our voice or something as a proxy for our thoughts, and some composition of models.

Sutton would say:

Yeah that's fine, but if we had the limitless CPU cycles/adequate technology, the solution of probe -> CPU would be better than what we develop.

My reasoning is simple, there are a whole class of problems that require embodiment, and I assume ASI would be able to solve those problems.

Regarding

> Point 1 is a big assumption. I am also not you, and although it's true that I have different goals, I share most of your human moral values and wish you no specific harm.

Yeah I also agree this a huge assumption. Why do I make that assumption? Well, to achieve cognition far beyond ours, they would have to be different from us by definition.

Maybe morals/virtues emerge as you become smarter, but I feel like that shouldn't be the null hypothesis here. This is entirely vibes based, I don't have a syllogism for it.

But if you assume that we have created something that is agentic and can reason much faster and more effectively than us, then us dying out seems very likely.

It will have goals different from ours, since it isn't us, and the idea that they will all be congruent with our homeostasis needs evidence.

If you simply assume:

1. it will have different goals (because it's not us)

2. it can achieve said goals despite our protests (it's smarter by assumption)

3. some goals will be in conflict with our homeostasis (we would share resources due to our shared location, Earth)

then we all die.

I just think this is silly because of the assumption that we can create some sort of ASI, not because of the syllogism that follows.

(As an intuition pump, we can hold on the order of ones of things in our working memory. Imagine facing a foe who can hold on the order of thousands of things when deciding in real time, or even millions.)

- regurgitate entire passages word for word, until that behavior is publicized and quickly RLHF'd away

- rip github repos almost entirely (some new Sonnet 3.5 demos Anthropic employees were bragging about on Twitter were basically 1:1 to a person's public repo)

It seems clear to me that not only can copyrighted work be retained and returned in near entirety by the architectures that undergird current frontier models, but the engineers working on these models will readily confuse a model regurgitating work to be "creating novel work".

On the other hand, doing an extensive search amongst possible head spaces you can occupy is a no-brainer for a consciousness implemented on a primate.