I.e. any task comes with the implicit meta task of amassing the resources and the power to do the task in the best possible way.

The point is made very well in the AI drives paper by Omohundro.

I didn't believe that myself at first but I've come around to it. I don't think it's inevitable though, but I think it's quite convincing that this tendency would naturally emerge through reinforcement learning.

This is a clear step change, I'm genuinely amazed.

Its answers are to-the-point, logically structured, clearly written. Like a really well made mini essay on your question.

It also really likes to build little simulations / models that you can tweak the parameters on. Amazing to explore how different assumptions affect economic scenarios, for example.

It's also great at picking (sourced) illustrations.

I'll be using this for everything that isn't coding from now on.

Secondly, in practice consciousness is rarely the criterion we use to make such moral judgments. Many people would think it's plausible that animals are conscious but still be fine with eating them.

It makes more sense to ask whether AIs have the ability to suffer. That's much more interesting, because we should be able to look into the models and discover 'neural correlates of suffering' - since they are artificial models, it is possible to specifically suppress that in training.

Even if you don't actually believe that suffering in AI is morally relevant because it wouldn't be consciously experienced, it seems that would be a good idea. Do you as a user really want to interact with an AI that gives you the feeling that it might be suffering in response to your prompts?

I think we don't really understand why deep learning works as well as it does, the thinking around that is, as far as I can tell, mostly a collection of empirical observations.

A fundamental theory of learning that can be used to predict optimal network architectures might enable smaller models that consume less energy.

Tao is speaking of a very particular kind of mathematics, that done out of pure curiosity.

But maths, even at the highest levels, often finds applications sooner or later.

It will be economically impossible to justify boycotting correct mathematics that no humans understand on grounds only of purity.

This may happen very soon: one of the obvious applications of novel mathematical results is in building stronger AI models.

The argument goes that livestock have a capacity for suffering, but killing them for meat without causing them suffering is ethical.

You can disagree with the position, or with its implementation in practice, but it's a consistent position in principle.

The business that had those problems didn't become Apple.

I thought the Bengio blog post posted here the other day answered that question quite well (ignoring your timeline).

The argument is that in training for the capability to achieve certain goals, you may inadvertently also train for secondary instrumental goals you did not intend, such as aggressive behavior or deception.

I'm actually no sure I buy that this is particularly likely, or necessary, but I found that take reasonable and worth considering.

This is not something that's just true on its own; it's on us as a species to make sure that it remains that way, for the sake of our dignity.

However, this position will be extremely difficult to defend, against the economic value of not caring.

It's the small teams you need to pay attention to, the organizations that really were limited by engineering capacity. These are by definition also less visible - for now.

E.g. I've gotten pretty good at git thanks to AI because it forced me into more complex workflows. I still can't remember the commands but I really grok the concepts and terminology on a far deeper level than before, where I made a conscious effort to stay on the happy path lest I get into some odd state I don't understand. Pretty sure those "odd states" would seem quite clear to me now.

Yes, true, but that's also very much a thing you can do something about by, well: reading books.

It's a technique we must preserve for the good of our culture, but, also, in a society full of people who have forgotten how to, an individual superpower.