That is depending on your threshold of certainty. 1% is not that high considering that, according to the OJJDP, 5 milion people were arrested for serious charges in 2019 so with a 1% false positive rate that would be 100,000 people falsely imprisoned every year.

It does not have to be and really can't be 0% but 1% is unreasonably high in my opinion. If it can't be helped then it can't be helped but that isn't necessarily the case with these devices.

For example, it might take several paragraphs to wholly capture all the meaningful information in one image in such a way that it can be reproduced accurately. Humans, and many animals, process large amounts of data before they are even capable of speech.

The data GPT-3 was provided with pales in comparison. It is unclear whether these GPT models are capable of induction because it may be that they need more or better sanitised data to develop abstract models. Therefore they should be scaled up further until they only negligepbly improve. If even then they, still, are incapable of general induction or have inaccurate models. Then the transformer model is not enough or perhaps we need a more diverse set of data (images, audio, thermosensors, etc).

It is true, however, that as the supply increases the likelihood that some consumers (namely addicts) won't be able to purchase any future batches increases as well (due to overdose, loss of income, etc). The market will likely reach an equilibrium eventually. Increases in supply have been met with increases in sales so the equilibrium hasn't been reached. This hasn't been a relevant consideration so far and there is no indication that it is now relevant. As such, it is hardly irrational or dumb to grow poppies at the moment.

The morality of the practice is another concern entirely. It is notable, however, that they do have legitimate medicinal applications.