I feel like this should be the default. Responsible disclosure to the affected company, followed immediately by disclosure to every politician in the dataset. Once we start collecting high profile cases this way instead of waiting X days for a faceless corporation to release a fix, companies will think twice about their security and the data they collect if that could make them end up on the shit list of the local government.

If you sit on an oil well, perhaps. In practice every static part of the production and distribution chain is a centralized target: from wells over refineries to depots, roads, harbours and finally power plants. You simply can't run fossils in a distributed local-first approach. That's also the reason why even for a massive oil producer like Russia, Ukraine can cause billions in damages with drones costing a couple hundred grand.

If you spread such a system out over dozens or hundreds of tiny chips, you'll be wasting most of its resources and lose hard to anyone who built a single chip setup.

The whole point of my argument is that neither is good, but from a technical perspective logprobs is probably the worst unless you train a model on specific outputs. In which case you'd throw out the generality again, so when I think about it more, it's actually the worst overall. In my experiments, having the model simply assign "high" or "low" probability in a structured output generally performs best. You can try numbers, but you will never get anything close to what you could expect from traditional ML. And most certainly not from logprobs.

I've found that using structured outputs solves this problem much better. Instead of letting a model generate only "A", "B" or "C" and looking at the probs, have it directly generate "Legitimate", "Spam" or "Phishing" or any other pre-defined option from a set of multi-token sequences. Behind the scenes it boils down to something quite similar, but you're not running into the risk that the model actually wanted to say "A phishing attempt seems likely, so answer (C) is correct.", which would lead "A" to have the highest probability in the first token. You can even use a reasoning budget this way either via inherent reasoning or a free-form part preceding the remaining output structure. You can also have it assign probabilities (either in words or numbers) using more complex output structures, but I would not rely on them much more than the token logprobs (they can still be quite good though).