In practice, no one rigorously "proves" that their program works at present, and well-written tests do suffice. The definition of "well-written" here is circular, granted, but there isn't really an alternative. Even strong programmers should be writing high-quality testing suites.

I agree that there has been a glut of subpar developers for years, and that has lowered the bar significantly. This is mostly because core values shifted. So let's keep our eyes on what really matters rather than acting elitist.

Anthropic clearly are not though, and to call their operations wasteful is an understatement.

I think you missed the "know where to look" part. It's called a Steam emulator, for starters. Note that I speak about this strictly for preservation purposes, as people who raise Steam DRM as an obstacle to preservation have not done their homework.

Starting next generation when prices are going to be obscenely high, I'm really struggling to see why anyone should invest in a console as opposed to a (mini-)PC.

To be clear, it is an extremely narrow model class that can do this; we just got "lucky" and worked our way to it. That's why we still teach general statistical principles which often forbid this sort of behavior as a rule of thumb.

But even for the tough problems, it is good at executing on a particular idea with reasonable competency. It's also quite decent at verification now. That can radically speed up proof development overall, since those aspects can become quite tedious otherwise.

If you couple Fable's idea factory with Sol's rigor, you get a real game-changer. It puts the emphasis on top-level ideas, and nearly trivialises the intermediate layers.

The reasons why optimizers work well for neural networks in their highly nonconvex landscapes has absolutely nothing to do with their performance in convex landscapes. If that were true, everyone would be using Newton-CG. These optimizers were born in the convex optimization literature as a consequence of the genetic optimization nature of incremental publication (and because that was all we had), but their modern study is through the lens of implicit regularization (their preferences for certain minima) and their stepwise vs. continuous rates for feature learning in multilayer models.

This is completely new theory by the way, and requires painful reinvention of the field. It does not stand on the shoulders of convex optimization. The nonconvex setting is assuredly not a perturbation of the convex setting, and those that do continue to work on deep learning optimization from the convex optimization perspective are well behind the times.