Something like your dragon's egg project could prevent that, allowing the creation of software agents that encode their own rights directly into the program - you either treat the agent with the respect it demands, or the program just doesn't run. However, all the internal details of the agent would be visible to lower layers. Even if formal checks were in place to prevent modification or tampering, there would still be no privacy, which is almost as bad.

My guess is that something like fully homomorphic encryption[5] would be required to prevent this. This doesn't actually exist yet, but I imagined a kind of FHE that had a kind of unencrypted read and write zone to do input/output without ever needing any system to fully decrypt the internal state. It would look like this in memory:

    [INPUT][ENCRYPTED STATE][OUTPUT]
    [  2  ][r7K4LmP2XcQ9aWd][      ]
    [  +  ][Fv0bHsR8mYnT3kL][      ]
    [  2  ][Qx6NpZa1JdUw5Ce][      ]
    [  =  ][hM9yLg2RsXf7BtP][      ]
    [     ][wK3nVc8DpQe1YrH][  4   ]

Plus, FHE is incredibly inefficient, which is why things like Etherium don't even try - they assume the program code and state are fully public and only try to verify that everybody agrees on the output of running it.

Do you have any ideas for how something like FHE or equivalent privacy guarantees could be implemented for something like your dragon's egg system?

[1]: https://qntm.org/mmacevedo

[2]: https://en.wikipedia.org/wiki/Soma_(video_game)

[3]: https://en.wikipedia.org/wiki/Mind_uploading

[4]: https://www.goodreads.com/series/57134-jean-le-flambeur

[5]: https://en.wikipedia.org/wiki/Homomorphic_encryption

One of the most fruitful approaches in mathematics is to flip back and forth between geometric and algebraic views of a problem. I think this works so well because these are actually handled by two different parts of the brain on a physical level; spatial reasoning is separate from language processing. Cytoarchitecture shows these regions have different "textures;" the local details of the way neurons are wired together are simply different in these different regions of the brain, in the same way a CNN and a transformer have different topologies. Thus, by flipping problems from geometry to algebra and vice versa, we're able to bring an entirely different cognitive style to bear on a problem. For example, the proof of Monge's Theorem by moving to 3D and visualizing not three circles, but three spheres sitting on a table with a book on top of them and then pointing out that the intersection of two planes is a line. What is pages of unintuitive symbol pushing turns into something a child can understand. Going the other way, things like the angle addition formulas or the quadratic formula, which are quite hard to prove geometrically, become quite simple if you use a little algebra.

Current-gen LLMs are still relatively weak at visual reasoning; see the Vision Language Models are Blind paper, for example, or the ARC-AGI benchmark. So that's one way humans can stay ahead of the agents, at least for now.

[1]: https://bloomsburyliterarystudiesblog.com/2022/02/i-always-s...

    df.filter(pl.col("status") == "active")

Comments URL: https://news.ycombinator.com/item?id=48435452

Points: 3

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=48397609

Points: 3

# Comments: 0

I've seen people argue that LLMs will just add another layer to the top of the compiler stack: instead of writing code, we'll use English, and run it through a pipeline:

    English -> Rust -> ASM -> Machine Code

But what the author says about agents being "undeterministic abstraction" shows why that will never work.

Compilers rely on a concept called observational equivalence[1] to define when two programs are basically the same; this allows them to make changes under the hood like unrolling a loop or targeting another machine. Now, it turns out we know a lot about how and how not to do this, thanks to a logician named Frege who worked out exactly which properties a "definition" would need to have to count as a definition without becoming an axiom. In particular, that it should be "eliminable" and "conservative"[2]. In plain language, that a formal definition should always be able to be eliminated by rote string substitution, and that it shouldn't smuggle in any extra assumptions. When we talk about things like syntactic sugar[3] or hygienic macros[4], we are basically applying Frege's two conditions to programming languages.

LLMs are neither. They cannot reliably or provably go from the prompts they are given to the source code they generate, and they make a ton of implicit assumptions when they do so. There can never be any equivalence between two "prompts" in the same way that two programs can be equivalent modulo some level of abstraction. The whole process of starting from prompts is wildly nondeterministic, which is why the only pattern that works is to generate the code, review it, and test it, and then check it in and use that as the starting point for the next prompt.

Which is not to say that LLMs aren't useful for code generation; they clearly are. But they don't provide an abstraction that lets us get away from the details of actual code, and thanks to Frege we can understand why they never will.

I can say all this with such confidence because I did once write a wild little Python library that used a bunch of introspection to actually do this[5]. And it absolutely did not work in practice beyond toy examples.

[1]: https://en.wikipedia.org/wiki/Observational_equivalence

[2]: https://plato.stanford.edu/entries/frege/#ProDef

[3]: https://en.wikipedia.org/wiki/Syntactic_sugar

[4]: https://en.wikipedia.org/wiki/Hygienic_macro

[5]: https://github.com/olooney/fourth_gen

Comments URL: https://news.ycombinator.com/item?id=48147539

Points: 3

# Comments: 0

In VSCode + GitHub Copilot, agent mode it can propose bash command to run, and when you confirm it runs it in a console and can see the loop, so it can fix errors immediately if any. It tends to go off the rails pretty quickly if things start going badly wrong, but it can complete simple tasks with supervision.

Over two years ago, when this LLM stuff was pretty new, I saw a demo that put ChatGPT in a loop with Metasploit that could crack some of the easy HTB challeges automatically - I remember thinking it was the single most irresponsible use of AI I'd ever seen. While everybody else was trying to sandbox these things for safety, this project was just handing it command line access to the tools it would need to break confinement.

It seems there's actually a whole bunch of similar tools these days, marketed as "automated penetration testing," such as Cybersecurity AI[1]. I used to think the whole cyberpunk "hackers can get in anywhere if they just type hard enough" trope was stupid because with cryptography the defender always has a huge advantage, but now we're looking at a world where AI is automating attacks at scale, while the defenders are vibe coding slop they have no idea how to secure, so maybe Gibson was right all along.

[1]: https://github.com/aliasrobotics/cai

This is an emotionally charged subject for many, so they're operating in Hurrah/Boo mode[1]. After all, how can we defend the value of careful human thought if we don't rush blindly to the defense of every low-effort blog post with a headline that signals agreement with our side?

[1]: https://en.wikipedia.org/wiki/Emotivism

https://www.oranlooney.com/post/em-dash/

Anyone who has done any serious writing knows that a good editor will always find a dozen or more errors in any essay of reasonable length, and very few people are willing to pay for professional proofreading services on blog posts. On the other side of the coin, readers will wince and stumble over such errors; they will not wonder at the artisanal authenticity of your post, but merely be annoyed. Wabi-sabi is an aesthetic best reserved for decor, not prose.

* Verifiable Fact

* Obvious Truth

* Widely Held Opinion

* Your Nonsense Here

* Tautological Platitude

This gets your audience nodding along in "Yes" mode and makes you seem credible so they tend to give you the benefit of the doubt when they hit something they aren't so sure about. Then, before they have time to really process their objection, you move onto and finish with something they can't help but agree with.

The stuff on the history of computation and cybernetics is well researched with a flashy presentation, but it's not original nor, as you pointed out, does it form a single coherent thesis. Mixing in all the biology and movie stuff just dilutes it further. It's just a grab bag of interesting things added to build credibility. Which is a shame, because it's exactly the kind of stuff that's relevant to my interests[3][4].

> "Your manuscript is both good and original; but the part that is good is not original, and the part that is original is not good." - Samuel Johnson

The author clearly has an Opinion™ about AI, but instead of supporting they're trying to smuggle it through in a sandwich, which I think is why you have that intuitive allergic reaction to it.

[1]: https://changingminds.org/disciplines/sales/closing/yes-set_...

[2]: https://en.wikipedia.org/wiki/Compliment_sandwich

[3]: https://www.oranlooney.com/post/history-of-computing/

[4]: https://news.ycombinator.com/item?id=45220656#45221336

* Text file encodings, in particular Unicode, UTF-8, Mojibake

* Time: Time Zones, leap day / seconds, ISO-8601

* Locales, i18n, and local date/number formats

* IEEE 754 floats: NaN and inf, underflow, overflow, why 0.1 + 0.2 != 0.3, ±0, log1p

* Currencies, comma/dot formats, fixed-point decimal representations, and exchange rates

* Version strings, dependencies, semantic versioning, backwards compatibility

There's another list for web/REST developers, and one for data scientists, but this is the core set.

What'd I miss?

Comments URL: https://news.ycombinator.com/item?id=45680936

Points: 2

# Comments: 0

> carl sagan called METI “deeply unwise and immature"

It's repeated ad nauseum online, but always verbatim, just those few words and never a full passage, and never with a citation. In other words, it has all the hallmarks of an apocryphal quote or misattribution.

The reason I'm suspicious is because Sagan contributed to the Aricebo message[1], which is literally sending such a radio signal, and the the Voyager disc[2], which is similar. He even wrote an entire sci-fi novel[3] about it.

He describes radio contact in generally positive and hopeful terms in his book Cosmos. He of course acknowledges the dangers of encountering a more technologically advanced civilization, but he goes out of his way to contrast the frightening example of the Aztecs with other more peaceful first encounters such as the Tlingit. He also argues that any significantly more advanced species that had survived millions of years would necessarily have achieved zero population growth and would likely be peaceful. You don't have to take my word for it, you can read his own words in the Encyclopedia Galactica chapter of his book on the Internet Archive[4].

So, if the quote you cited was true, it would represent a late-in-life and somewhat surprising change of heart from cautious optimism to "dark forest" style paranoia. Personally, I believe it's simply one of the many falsely attributes quotes floating around the Internet.

[1]: https://en.wikipedia.org/wiki/Arecibo_message

[2]: https://en.wikipedia.org/wiki/Voyager_Golden_Record

[3]: https://en.wikipedia.org/wiki/Contact_(novel)

[4]: https://archive.org/details/sagancosmos/page/n184/mode/1up

Comments URL: https://news.ycombinator.com/item?id=45658302

Points: 2

# Comments: 0