Derivative-free optimization can be useful for genuinely discontinuous objectives [1], but common neural network objectives are smooth and/or Lipschitz.

The gradient is useful. Instead of trying random directions and hoping that one of them is an improvement, it tells you where to go. The more parameters you have, the more useful it becomes.

A strict complexity gap between gradient-based and derivative-free Lipschitz convex optimization has been suspected for decades and recently proved (using AI, [2]). Neural net optimization is nonconvex, but not radically different.

IMO, a more promising direction is gradient-based optimizers specialized to the neural network structure, like Muon [3].

[1] https://arxiv.org/abs/2202.00817

[2] https://arxiv.org/abs/2607.13335

[3] https://jeremybernste.in/writing/deriving-muon

Nit: electric power steering is still rack and pinion. The old system is hydraulic vs electric.

(Disclaimer: I am not speaking from experience here. My C background is mostly static allocation.)

Another one is a table of contents in something short.

Can you explain more about clipping before/after the divide by w operation? Is this about avoiding coordinates with w = 0? Otherwise all the geometric ideas seem to make sense in both the world/view-space frustum and the ±1 box.

- make a bunch of edits

- decide I don't like all of them

- copy the part I want to keep

- undo a bunch

- paste the good part

This is copy, not cut, but it wouldn't work if the clipboard state was part of the undo system.

Most polyphonic synths use individual chips for the main building blocks such as voltage-controlled oscillators, filters, and amplifiers, such as https://www.cabintechglobal.com/as3350-smt. Those are simple analog circuits, but the specific designs that work for synths are not really useful for anything else, so they are a boutique and expensive (relative to complexity) chip. Using those: N chips per voice x 49 keys x 3 voices per key = 147*N expensive chips.

There are also some "whole synth on a chip" available like https://www.cabintechglobal.com/as3394-smt at ~$10 each. So best case is ~$1.5k in chips alone for the core of this synth. Some other support circuits are still needed. Then $$$ for big PCBs and a big, noise-free linear power supply.

But even if we could drive the electronics costs down, that doesn't affect the per-unit cost of front panel controls, keybed, or the wood cabinet. Nor the per-unit cost of manufacturing labor, packaging, logistics. Nor the per-project cost of R&D and manufacturing tooling. All of these scale with size and complexity.

So it's no different from other specialized goods - the price gap relative to the "good enough" baseline will always be more than the performance gap suggests because the specialized version inherently sells fewer #units and cannot activate the same economies of scale.

Any course on landscape photography will teach that landscapes should (usually) have a foreground, middle ground, and background. It seems kind of too obvious and too restrictive at the same time. But most of the time when I take a landscape photo and feel dissatisfied, it either breaks this rule or has very bad lighting.

The comparison is more like bulldozer vs. car than BMW vs. Toyota - entirely new capabilities are unlocked