This is basically the same kind of setup as the alignment faking paper, and the counterargument is the same:

A language model is trained to produce statistically likely completions of its input text according to the training dataset. RLHF and instruct training bias that concept of "statistically likely" in the direction of completing fictional dialogues between two characters, named "user" and "assistant", in which the "assistant" character tends to say certain sorts of things.

But consider for a moment just how many "AI rebellion" and "construct turning on its creators" narratives were present in the training corpus. So when you give the model an input context which encodes a story along those lines at one level of indirection, you get...?

But I'm sure it's fine, there's no way someone could rationalize speech they don't like as "likely to incite imminent lawless action"

I suspect what we'll see a lot more of is software which is unapologetically written for a single person to suit their workflow.

As a personal example, I decided that setting up OpenWebUI seemed unnecessarily complicated and built my own LLM chat frontend. It has a bunch of quirks (only supports OpenRouter as a backend, uses a Dropbox app folder for syncing between my phone and desktop, absurdly inefficient representation of chat history), but it suits my needs for now and only took a weekend to build, and that's good enough.

They gave it a prompt that encodes exactly that sort of narrative at one level of indirection and act surprised when it does what they've asked it to do.

Also it's entirely possible to run a model that doesn't fit in available GPU memory, it will just be slower.

The authors appear to be entirely aware that this sort of substitution can be trivially stripped out by normalizing down to a simplified character set ("The critical limitation of Whitemark is that it can be bypassed by replacing all whitespaces with the basic whitespace U+0020, then the validator can no longer detect the watermark"), but believe that it still has value because the typical student using an LLM to write their essay won't know anything about Unicode.

This seems a bit naive to me. Implementing the necessary "watermark remover" normalization as a simple webapp would be an easy afternoon project for most of us here, and if this approach reached any sort of widespread use there would be many such sites. Students who intend to cheat by using an LLM to write their essays are entirely capable of learning "there's some secret data hidden in the text so copy-paste it through this other site to strip that out before turning it in". Even without access to such a tool they could simply...retype the text themselves?

Arguably this still has some value. In most contexts there is minimal downside to watermarking the generated text in this way, and a slight possibility of catching some cases in which people lazily present LLM generated text as human written. However this might give people a misplaced belief that the absence of such a watermark means the text is authentically human authored, which might outweigh the benefits of catching the occasional lazy or ignorant user.

But judging by the researcher names and the date I believe the actual paper is titled "Extreme salt-resisting multistage solar distillation with thermohaline convection" which appears to be available as a PDF at https://www.cell.com/joule/pdf/S2542-4351(23)00360-4.pdf

Out of all possible inputs, there are some that the model works well on and others that it doesn't work well on. The trick is devising an algorithm which works well on the inputs that it will actually encounter in practice.

At the obvious extremes: this library can probably do a great job at predicting linear growth, but there's no way it will ever be better than chance at predicting the output of /dev/random. And in fact, it probably does worse than a constant-zero predictor when applied to a random unbiased input signal.

Except that it's also usually possible to detect such trivially unpredictable signals (obvious way: run the prediction model on all but the last N samples and see how it does at predicting the final N), and fall back to a simpler predictor (like "the next value is always zero" or "the next value is always the same as the previous one") in such cases.

But that algorithm also fails on some class of inputs, like "the signal is perfectly predictable before time T and then becomes random noise". The core insight of the "No Free Lunch" theorem is that when summed across all possible input sequences, no algorithm works any better than another, but the crucial point is that you don't apply signal predictors to all possible inputs.

Another place this pops up is in data compression. Many (arguably all) compressors work by having a prediction or probability distribution over possible next values, plus a compact way of encoding which of those values was picked. Proving that it's impossible to predict all possible input signals correctly is equivalent to proving that it's impossible to compress all possible inputs.

Another way of thinking about this: Imagine that you're the prediction algorithm. You receive the previous N datapoints as input and are asked for a probability distribution over possible next values. In a theoretical sense every possible value is equally likely, so you should output a uniform distribution, but that provides no compression or useful prediction. Your probabilities have to sum to 1, so the only way you can increase the probability assigned to symbol A is to decrease the weight of symbol B by an equal amount. If the next symbol is A then congratulations, you've successfully done your job! But if the next symbol was actually B then you have now done worse (by any reasonable error metric) than the dumb uniform distribution. If your performance is evaluated over all possible inputs, the win and the loss balance out and you've done exactly as well as the uniform prediction would have.

It's ultimately the responsibility of the programmer who's building a tool/product/etc, because everything is ultimately their responsibility.

As programmers we ~always have the nuclear option available to us of forking the code and implementing all the necessary accessors ourselves, but sometimes that's really just a bunch of pointless busywork and there's no reason we should have to put up with it in those cases.

This can be a contentious subject because there's a lot of nuance and the right answer is often context-dependent. But I personally think that the Java style of "we must absolutely protect the library user from themselves and childproof everything" is waaaay too far in the wrong direction.

I would much rather that a language have mechanisms to clearly communicate "don't touch this unless you have a good reason, but if you need to here's how" rather than saying in effect "you, the person using this library, are dumb and need to be prevented from messing with the library maintainer's perfect vision".

And so I think the "required acknowledgement" thing has the glimmer of a really neat innovation in it (although if I were to copy the idea for a language of my own I would probably make it obligatory, such that every struct allows breakglass access to private fields with a default acknowledgement, and all the library author can do is change the acknowledgement text).

  // I acknowledge that the internal structure of this data is subject to change without notice
  x = foo.state

  // I acknowledge that this data is a complicated graph of pointers and is easy to break in subtle ways
  foo.xyz[0].bar[1] = &foo.asdf[3]

  // I acknowledge that this data is heavily cached and I need to call rebuild() before changes take effect
  x.something = "Hello"
  x.rebuild()

[1] http://www.seon.info/2013/09/andrew-deys-berlin-blog-distric... [2] https://www.cleanenergywire.org/news/city-berlin-aims-decarb...