Letting another LLM decide quality probably isn't the only meterstick you want, either.

It seems like a really interesting tool and maybe future models will know how to use these tools more effectively, but yeah I remain skeptical.

The more I think about it, I suspect that either the training data or the finetuning process (or both) will have to incorporate these tools to really get benefit from them.

https://unleadedkids.org/about-us/

Note that the state did eventually modify the communications that the law required.

https://oehha.ca.gov/proposition-65/crnr/proposition-65-clea...

Still, six months after it published its data, the FDA reapproved perchlorate for use in super sacks. Advocates say the agency used flawed science and manipulated the data. The FDA wrote that there was no meaningful increase in perchlorate across all the 250 foods it measured. But critics countered that the problem was not in all the food – the problem was in kids’ cereal.

The FDA largely ignored the cereal findings, and mostly does not consider endocrine system harms when developing regulations, said Tom Neltner, a former scientist with the EDF who is now director of the Unleaded Kids non-profit. He helped lead litigation over perchlorate, which was ultimately dismissed.

I would really like to see more policy (from whomever: local, state, federal, judiciary, legislative, executive, red, blue, green, black) that actually accomplishes some of the things that RFK Jr says that he wants to accomplish.