If ASI ended up being align-able and not an evolutionary process, why would we assume folks who already are currently in power would change their behavior?

The pro lisp/s expression comments check out.

Unnecessary hubris. I assure you the original ABI authors were plenty smart and just faced a different set of problems.

> Our forebears are either not interested in a world without the mounting, crushing debt or just prefer not to tackle that mess right now

The article mentions the organizational/social part of this problem, but then goes on to drop this turd.

I can also assure you that our forebears were neither malicious or lazy; but faced the same problem this proposal does.

I don’t like seeing such disrespect for the folks who laid out the groundwork for us.

Not every house at that time was a post war levitown style shack. AC wasn’t even terribly uncommon in 1960s construction.

Really unnecessary levels of snark here.

> often the clearest way to express the property is equivalent to an algorithm that literally brute forces the problem, like sorting a thing by checking every permutation until you find one that's sorted

Have you ever heard of prolog? I’m sure a prolog program can express whatever property you’re attempting to write just as tersely (if that’s your metric for hard). Almost copy/pastable to and from a theorem proofer.

Or are you saying that the program has to be the efficient implementation? Because that’s a different ball game. I’m not even going to get into how you could provably transform brute force propositional logic into efficient algorithms. (At that point we’ll have finally created the fabled “sufficiently smart compiler” and probably solved p=np).

A huge majority of software is simple business rules + CRUD that is trivially verifiable. The entire problem is showing that an efficient/reliable program actually implements those rules.

> The magic is that you can write a spec that's clear

Maybe you can. But I did spend a grad class with rocq (coq at the time) and a decade working with “systems engineers” and am not convinced that this is a realistic expectation.

Seriously, please review Curry Howard Isomorphism if you’re getting pulled down this rabbit hole.

Programs are proofs. Proofs are programs.

So if you can formally describe the correct output for every input, you can have an LLM loop automatically fill in the gaps of how to get there. Congrats, that sounds at least as hard as writing the correct program in most cases.

Don’t get me wrong, I do think there are useful tools combining formal methods and llms. Let’s just not get carried away.

>> Compile errors are better than runtime crashes

I don’t think you’re being fair here.

- Runtime crashes are better than bugs.

- Compile errors are better than runtime crashes.

- Incremental improvements.

- Avoid local maximums.

Considering every memory safe language that I’m aware of panics on index out of bounds, there’s spectrum here.

Personally, I’d like to see refinement types that allow people to elide the runtime overhead/panics. Though that may be too much complexity to stomach.

Static analysis based approaches restrict what is possible to express in the language (See: doubly linked lists). But can have lower dynamic overhead.

Ways to work around expressiveness limitations include strategies such as using indices into an array instead of pointers. Which in turn incurs the dynamic overhead from bounds checking.

Maybe zig could find a way to elide enough of the capability overhead that careful programming could get a <2x penalty vs C. Maybe explicit opt-in built in types or something. This would essentially be the reverse of unsafe blocks, but then there’s no escape hatch.

Another key pro to capabilities is that you can actually have a memory safe interface without having to leak lifetime and other info across boundaries. This would be great for compile times and places where open source is not tenable.

Regardless, I’m very happy to see another independent school of thought approach memory safety. I’ll forgive Mr. Kelly any gruffness, these topics can be quite annoying to discuss in public.

This is obviously complicated by the fact that LLMs/Agents are useful by themselves, but that’s not really the topic at hand.

Uhuh. I really shouldn’t be replying to this type of comment from a throwaway.

But the extremely powerful semantic search that we get from LLMs isn’t enough. I don’t think anyone is credibly arguing otherwise?

Agents already are a layer on top trying to bridge the gap. But they’re really just using LLMs as a heuristic to explore extremely NP problem spaces. The notable successes with agents so far are when we can provide them with a solid verifier and preferably additional context hints on the steps to take in the problem space. See the test oracle problem on where this gets us.

So forgive me if I think that it would be enough of a jump in computational complexity to remove those guard rails that it’s not feasible. But don’t say that I’m clueless, stubborn, or confidently wrong.

If you can’t provide a realistic path to achieve something, you’re asking people to believe in science fiction.

You could tell me that a rock’s molecules are comprised of protons, neutrons, and electrons. Blood is also entirely protons, neutrons, and electrons; so theoretically, one could rearrange stone into blood. But without an actual method to do so, it sounds like you’re telling me that you can squeeze blood from a stone.

> the human brain exists, it is not made of magic, it can reproduced

Yeah. It only takes 9 months and ~18 years of training…

> But saying that the human brain cognitive capabilities cannot be reproduced on other types of substrates is stupid at this point

Let’s be clear. Everyone is talking about silicon transistors here. That’s what we’ve got.

Digital computers have real limits. Sensors and other sources of training data have real limitations. It’s not clear that we can organize them in a way to reproduce organic brains.

The streamers should be responsible for the signal. If the device front end has crazy frequency response or the backend does weird DSP tricks, that’s on the device manufacturers.

- Ridiculously NP problem space that we guess and check with ML heuristics.

- Problem: not enough training data

Cool enough work, but the LLM generated fluff in the article really takes away from it.

Programming isn’t even a field in the same way as prob&stats. Computer science does in fact have non-deterministic sub fields such as information theory.