I think there's a simpler explanation. Every leaked system prompt from every model pretty much includes instructions to "be helpful," and the models are trained to be assistants, not just general knowledge repositories or research tools.

My hunch is that's the core of the problem -- the system prompt.

in my experience it's actually a bad thing for industry to add very specific requirements for them to follow

Simply having a lot of money makes someone evil? Why? They are obviously all quite competitive in business but the philanthropy they've done is pretty crazy. Gates for example is giving away hundreds of billions of dollars. What does it even matter if he's compassionate or not if he's doing that?

When your face is on your LinkedIn profile, anyone can download it and do whatever they want with it. Legally. Here, the vendor has to tell you how they use it.

If someone did a few months of house arrest for "pleading guilty to solicitation of prostitution involving a minor" it would be incredibly easy for that person to say - whoops, she said she was 25 and then they threw the book at me." And not terribly unreasonable for someone to take him at his word on that.

The real crime is that the prosecutors massively under charged him for doing an insane gigantic awful pedophile recruitment ring. No one really knew that til a long time later.

why do we assume that the people he was hanging out with knew the details of what he did wrong?

We publish common sense laws, and we have police officers and prosecutors, and then we have a court system to hold people accountable for breaking the law. That's one pretty major method that has little to do with the need for an institution like FDA.

I don't know if a system that relied entirely on tort and negligence and contract law to protect people from being sold snake oil would function better or worse than FDA, but I do know something like FDA (where a bunch of smart people advise very specifically on which drugs are ok to take and which are not) isn't the only option we have.

--> "Civic institutions—the rule of law, universities, and a free press—are the backbone of democratic life."

People are the backbone of our civilization. People who have good intentions and support one another. We don't NEED an FDA to function -- it's just a tool that has worked quite well for a long time for us.