The particular ideology of Christianity was super effective at laying the groundwork for modern civilization. Of course, these ideals have since evolved and have since become some kind of secular ideology. Granted, it was imprecise to attribute a fear of death to Christianity. But the sanctity of life is in practice a close cousin of a fear of death. Modern secularism having no metaphysical alternative has made this fear explicit.

No, I do not think human autonomy should be sacrificed to defeat death. I personally do not fear death. It's as irrational to fear as it is to fear a deep sleep. I do fear dying painfully. Tragically, we as a society resist alleviating the pain associated with the process of dying (by speeding it up) for the same reasons. If only we could learn to take our principles in moderate doses.

The question is in what context does the human psyche thrive? We find meaning in personal struggle and overcoming obstacles. We're happiest when we're striving for something out of reach but seemingly attainable. We're much less happy when we have all of our needs met but have no source of meaning or fulfilling struggle. Our evolutionary history is one of struggling against hardship and finding personal meaning in the process. This is a core feature of our psyche and it is to our detriment to eliminate it.

This isn't to say we should all be struggling for our next meal. Much like cats, our core drives can be exercised and satiated through a simulacrum. There are many ways it can be satisfied, but strive we must, against something with inherent meaning to us. The problem with AI fully realized is that it takes away most avenues for human achievement, thus the struggle loses all meaning.

It's funny, the justification for accelerationism is that people can be paid (UBI) to pursue their passions. On the other hand, we have those that are asking why anyone should be paid for work that isn't economically viable. I suspect we're being sold a load of bullshit.

Is solving cancer worth ceding human meaning and human autonomy to the machines? I haven't seen anyone ask this question but it seems the central issue of this revolution. Personally, I don't think it is.

Perhaps machines will lead to a perfectly homogenous society completely absent of negative valence whatsoever. But if every experience is also devoid of any personal meaning, what's the point? The optimal human state is not one of peak hedonism, or wholly absent of suffering.

It's quite bizarre how many people feel so personally affronted at the existence of a rather exclusive intellectual community that they take glee in seeing it decimated overnight. But make no mistake, AI is coming for the things you value as well.

Whether some trait is emergent shouldn't depend on the state of our knowledge. That no one predicted how far LLMs would go is a demerit for our model of them and doesn't imply they have extremely complex "emergent" interactions. My conception of emergence is asking whether something can be "easily" predicted from an ideal understanding of the component parts. Simple/linear aggregations of the correct semantics of precursor neurons is what is at stake here. In other words, is understanding the semantics of the precursor neurons the vast majority of the hard part in understanding what comes next? Past failures of prediction do not bear on this question much at all.

An example from the article that is informative to my view is the association the network made to the base64 encoding of plain text substrings. If you had asked me, I would have assumed the network decodes base64 live as it traverses the layers. Instead (at least some of) these associations are precomputed during pretraining and recorded in the form of shared affinity in neurons. Active computation and precomputed associations are on two ends of the intelligence spectrum. The human brain is strongly biased towards precomputed associations. It seems that LLMs are as well. The more I see about their inner workings, the more I suspect they operate mostly in the form of these explicit associations. The computations they actively manifest across the network can't be that complicated due to depth constraints. There just aren't enough compute steps[1] for the kinds of complex, non-linear, unpredictable dynamics you get from compute heavy regimes.

>Something doesn't cease to become emergent if you can roughly explain it by making references to its parts. We can broadly explain how human behaviours result from drives + sensory processing + associations and etc.

When I talk about explaining some phenomena, I mean it in a robust sense that e.g. gives you the capacity to predict downstream behavior with high accuracy.

[1] These discussions always seem to have difficulty distinguish the computation that realizes the system from the computation the system as such engages in. There is a huge amount of compute that realizes the feed forward pass of an LLM. But the computational steps directed by the LLM to resolve its output from the prompt is severely restricted by the layers in the forward pass of the network (or however many virtual layers from looping).

I doubt my usage is idiosyncratic. Rather, the term is so ill defined that people often use it inappropriately. Emergent means something along the lines of not easily predictable behavior from the behavior of the component parts. My claim is nothing in LLMs is emergent in the sense of being inexplicable from an ideal consideration of simple/linear aggregations of its neurons. This isn't something I can prove, but it certainly isn't obviously false from just considering recent mathematical results. Higher order/non-linear interactions are characteristic of this kind of inexplicable behavior from complex interactions we characterize as emergent. But there are good reasons to think this kind of non-linearity in any semantic regime is next to impossible in principle.

Semantic geometric representations are characterized by linearity in response dynamics such that a linear change in one variable results in a linear semantic change in related variables. That's how vector embeddings capture semantic relationships. Call it the linearity hypothesis. Non-linear dynamics in the embedding space just destroys semantic relationships in most cases. There is no way to keep an entire network of relationships sufficiently correlated if non-linear response dynamics is a part of the representational toolbox. Non-linear dynamics in general kills correlations.

Testing this against recent LLM proofs, what would need to be shown is that the relevant neural activations do not decompose the semantic content of the proofs (or the reasoning trace) in a straightforward manner, but rather the reasoning trace is unpredictable from the semantic content of the activated neurons that precede the tokens of the reasoning trace. But this idea strains credulity; it is almost inconceivable that this would be the case. We should be highly incredulous of the idea that complex non-linear interactions can be wholly responsible for the long-running coherent output in LLMs. The default hypothesis should be that there is no emergent behavior (as I defined it) in neural networks.

Emergent behaviors in terms of seemingly random combinations of features producing unpredictable but coherent/intentional output basically don't exist. In other words, there are no "hidden thoughts" in an LLM encoded in higher order combinations of features. Inasmuch as non-semantic features of the input influence output, it does so in non-semantic ways that are degenerate in terms of the quality of output. The only thing emergent in LLMs is degenerate state leading to a breakdown in tracking relevant features of the input and a reduction in quality of output. But the relevance of this kind of degeneracy to LLM output is at the margins. Tracking activated state predicts subsequent output to a high degree of accuracy.

>but it isn't straightforward at all, and is generally very messy and you get all sorts of unpleasant sides effects (e.g. abliterated models not performing as well).

I expect this has as much to do with our lack of precision in our tools rather than with the concept in principle.

>Consider our publication earlier this year, in which we reported that 36 of 38 companies shipped pieces of the 1918 influenza genome. That was collectively enough to generate the virus using reverse genetics several times over – and they were ordered by a student using a pseudonym falsely claiming to work at an office address. No one asked whether we had permission to work with 1918 influenza (a select agent), or why we needed genes from it, even though the official U.S. guidance at the time instructed them to do so. In fairness to the companies, that guidance was voluntary, and the fragments we ordered were legal to produce and sell domestically in every nation of the world (export controls are complicated). That is still true today, which is why we have a problem.

Not mine either. But there is a vocal contingent of experts in biotech that have been sounding the alarm for the last couple of decades at the potential for rogue groups to develop weaponized pathogens for terrorism. And this was years before LLMs. Apparently constructing a sufficiently capable lab isn't really a bottleneck due to how freely available equipment is by mail order and how unsophisticated the processes are in practice.

There's an opportunity cost to terrorism just like with any time sink. The effort to build up knowledge enough to produce some weaponized pathogen will be compared to just doing traditional terrorism. For some low capability terror cell, its easy to see how the cost/benefit analysis has been in favor of traditional terrorism up to now. The kinds of terror acts that take years of sustained effort to execute are rare. But as the barriers to entry to bioterrorism fall away and become widely accessible we may see the cost/benefit shift.