The people described in the article aren't just "people who know who Yudkowsky is", they're self-avowed EAs and active LessWrong members.

The most important thing that a good design doc does is specify what's in and out of scope. The second most important thing is to precisely define common vocabulary - what are the important concepts in the problem you're solving, and how should they relate to each other? All of that information serves to ground the day-to-day work in what's important. I find myself starting every Claude session with "read this doc and get familiar with the world, then we'll get to work on a part of it".

(The same is true when working with humans, especially but not limited to junior engineers who aren't used to managing a project longer than a week or two. AI coding agents just never grow out of that phase.)

And the "why are they developing AI if they think it's so dangerous" argument is neither new nor persuasive. They're developing it because (1) they think the potential benefits are as great as the potential risks, and they know that if they aren't one of the actors on the frontier (2) they won't be able to propose meaningful solutions and (3) their opinions won't be taken seriously. Amodei is in rooms with powerful people to propose these things because Anthropic is successfully developing frontier models. The heads of various NGOs and advocacy groups that are concerned with AI safety but not themselves working on those systems are... nowhere, writing pamphlets and blogs that not you nor I nor anyone in Congress will ever read.

If I switched to spending more time on the hobbyist craft, I would pretty quickly have to give up doing well at the job. And the parts of the craft I enjoyed the most are the puzzles around distributed systems and high-scale correctness, which tend not to come up in the hobbyist context anyway.

In the ad marketplace there are four participants: the advertiser, the user, the network (Google), and the publisher (also Google for AdWords, other websites for AdSense and so on, and effectively the channel owner for YouTube).

In the click spam or botnet case, the bad actor is the user, who is usually associated with a publisher trying to get extra money (although not always - there's reasons like auction manipulation where some advertisers run click bots too). In the bad-ads case, the bad actor is the advertiser.

At Google, each of those problems has their own well funded team, but they do also share some data to help catch rings of bad guys.

Your "point 1" is literally the argument of the paper. If that scenario arises, the market collapses and no further sales can be made. That's the whole problem. As someone who takes a percentage of every sale, you want to keep the lemon-sellers out even though in the short run they make you extra money.

Your "point 2", if it's meant to be a rebuttal, isn't much of one. Buyers don't need to accurately know exactly what fraction of sellers are fraudulent; if they believe that it's 50-50, the same thing happens, even if the true rate is 80-20 in favor of good sellers. Conversely, if buyers are overly optimistic about quality, the market can persist despite a level of fraud that's higher than should be tolerated. But in any case, things like reviews and external reporting should eventually give them good information.

It's not that they're not trying, it's just a very hard problem.

In most states, employers have broad rights to fire employees at will, including no reason or "he wore the wrong color shirt". That doesn't mean that if you have a dickhead manager who fires you because he doesn't like the color of your shirt isn't a dickhead, and if you defended that manager I would similarly consider you unfit to participate in civil society.

I mean... "leave to renew" is a horrible policy, it's just one that we've normalized since 9/11. It's incredibly wasteful and disruptive, and serves no purpose. There's no legitimate reason why a long-term residential visa like an H1-B can't be renewed by mail or electronically. I can renew my driver's license by mail, and I'm a hell of a lot more likely to cause serious damage with a car than my foreign coworkers are by continuing to do their jobs well.

The reason the State Department hides behind, by the way, is that domestic offices don't have the ability to take biometric data (fingerprints), which is self-evidently pretextual.

But also, it's only been "the policy" since 2004. That's technically "decades", but only just, and it was also introduced for idiotic pretextual reasons. The ostensible reason is that domestic State Department offices don't have the equipment to capture biometric data. Seriously. That's the argument. The law doesn't say "you have to leave to renew", it says "you have to get fingerprinted to renew", and conveniently there's nowhere in the United States that can do that, despite every police station in the country having that equipment at this point.

Totalitarian, and trending toward failed state.

> US

Not eligible for the reason I stated: the US economy is much less than 10% petroleum. We have a lot of oil, but we have a lot more of everything else.

> gulf states

Mostly brutal, repressive dictatorships (especially Saudi Arabia and Iran) - some are better than others, but the trend is poor.

> Just because you don't like a government doesn't mean it's doing a bad job at, well, holding a state together

You're moving the goalposts a fair amount here. The thread started with "good governance", and I said "well run" - it's hard to say that the bar for either of those is simply maintaining superficial order. But even if you set that aside, consider the famous quote from the Shiekh of Dubai about the fragility of the petrostates that haven't figured out how to build strong civil societies with their wealth: "my grandfather rode a camel, my father rode a camel, I drive a Mercedes, my son drives a Land Rover, his son will drive a Land Rover, but his son will ride a camel."