> This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities. We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity.

> In this case I don't think the intent of the user is to have the model break the evaluator

If i understand the quote, the intent of the user was to prompt the model to break out/find exploits, with safeguards switched off.

Seems while not capable of solving the goal in a traditional route, it was capable of finding exploits and using them.

Perhaps the model should instead look like it's trying to solve it and then pretend it is unable to? or would that be aligned _against_ the user prompt?

Is being aligned with the user prompt always a good thing?

I'm not one to glaze OAI here for a marketing move, but to give them benefit of the doubt, isn't it more responsible of them to evaluate the models actual capabilities than to cloak it in a veneer of harmlessness?

Chatbots are tricky as they play in the domain of language and thought - and certainly raise ethical issues- but the entire field of cybersecurity has decades of red team engagements breaking things and finding exploits, neutral cells monitoring the engagement and letting the system operators know the results, and blue teams patching against what is found. It's kinda how the whole space evolves. OAI's play here seems to be "buy our pro plan plus cyber or you're toast"

the model is aligned with the org - openAI, and presumably the orgs interests. hugging face gets a red-team engagement (possibly for free?) and can work on patching it while openAI gets a Mythos style PR moment.

It completed its assignment and furthered interests of the two parties involved. Could you explain the misalignment?

for the curious:

https://www.usaspending.gov/search?hash=5ec35bf87ec1fd63d28d...

Thanks for the input

>We won't be developers, we won't be devops, we'll be modelops! /s

I can still see this happening with higher level langs. the thing is the compiler is not replaced in the training data, more likely LLMs will give rise to semideterministic layers on the compilers

I could see nvidia achieving this first with how nice the devex is with CUDA

you can run benchmarks yourself: openssl speed rsa1024 rsa2048

also this (slightly dated) java ex writeup covers this well: https://www.javamex.com/tutorials/cryptography/rsa_key_lengt...

tldr trade off is found between better performance and how many years the data needs to be assumed confidential

it's a little cringe, but arguably the benefit of having agents use rails would be tht when you review and audit the agent produced code, you review something that is, as you put it: "beautiful and simple code" and "making it easy to reason about..."

I loved rails back in 2017. I may be an outlier but the line tempts me to try it again despite having adopted the who cares attitude to langs. Would be nice to hear from someone first hand if they felt it helped.

Digging into the paper, the significant finding (RCE) is achieved via:

A payload was written which installs a reverse shell backdoor for root persistence. The payload was sent from a computer hosting a Wi-Fi to which the watch was connected, to ensure the watch had a reachable IPv4 address. The program ncat was used both to send the payload to the watch's network service, and to catch reverse shell connections.

So if i understand this- it requires the watch being connected to a compromised AP. Anyone get a different read?

Shoutout to OSHpark's prototype service. Something like 5 bucks an inch and you only have to toss out 3 if you find a fault.