Helping people deal with bad UX.

Everybody who has worked in a traditional corporate environment has encountered poorly tutorialized, poorly labeled, counterintuitive user interfaces where the only way to know how to get something done is to have somebody sit over your shoulder and tell you which buttons to click.

Oh, you want to figure out which OIM entitlements your coworker has that you don't to debug an access issue? You just need to know that you have to click the "compliance" button to get there. Why is it called "compliance" when what it actually does is let you export comparisons from the user tables? Because fuck you that's why. Memorize it.

An AI assistant that can read the procedures for you, or sometimes even read the source code, or the website code, can show you how to accomplish something that the UI on its own fails to do.

It makes sense to not trust AI models with the ability to read and alter your screen for many many many reasons, but the developer of the tool knows that as well. A tool that lets your ai do something doesn't also let it do anything.

So, as always, it's a question of if you trust the software producer. You're giving them the ability to draw on your screen too. Would you have the same fear if there were no AI involved?

For stuff like this I imagine AI misbehavior as a novel failure trigger not a novel failure type. It can only do the harm that the software was able to do on its own already.

You can make your artificial sunrise take as long as you want, and connect them to a sleep tracker so they're timed to wake you up during a comfortable phase.

I've been doing this for 12 years, and I don't think I have ever been upset to hear my alarm since, as long as I'm home and the lights have prepared me. Lately I've even been waking up before my alarm about half the time, even though I work on a schedule one time zone east of where I am.

It's shocking to me that it's not more popular for everybody. I guess the benefit is hard to explain.

Maybe it's just me.

For like, 90% of conversations, I don't want it to let technical inaccuracies and rhetorical flourishes slide. I want it to tell me that the point I'm making is technically wrong because an expert would recognize subtle misuse of terminology, or because there's an exception or edge case that I didn't proactively insert as a caveat, so that it is my decision to ignore that advice and be a little wrong on purpose to suit my writing goals.

What I don't want is for the AI to assume my writing goals, and be incorrect because it believes that is what I want. I want it to "well ackshually" me so I can say "shut up, nerd".

Like, there's another comment in this thread that I ran by claude to check my understanding about today's post-training methods and how they avoid sycophancy, and claude responded by splitting a bunch hairs over like, "well, technically this is still RLHF, its just that there's other feedback signals mixed in, and the preference is detected in other ways, and ai judges are involved as a filter for examples, this and that and blah blah blah". Shut up, Nerd. In the context of this conversation, RLHF is already being used as synecdoche for user preference feedback, readers understand that, and even if they don't, their misunderstanding is completely harmless. I will not be taking all the wind out of the sails of the point I'm trying to make inserting your three paragraphs of irrelevant clarification in the name of technical correctness, thank you very much.

As long as receiving nitpicks and technical minutiae implies 1. there are no larger structural problems and 2. the model isn't rolling over to please me with sycophancy, I figure this is ideal.

I think there's a second effect that's a selection bias for the experiment itself and probably cuts against the supposed dangers of this result: a chatbot can tell you things only after you have asked it a question.

There is one thing that every single study participant (including the uninformed reddit users) have in common: they all knew that the person or entity was trying to convince them, and they read the words and thought about them enough to craft a reply.

This means that all of the people here were available to be convinced and self-identified as such.

Being open minded is work. You have to doubt your own beliefs, you have to disregard evidence that you previously found convincing, you have to crank up the empathy to put yourself in another's shoes, you have to listen intently to understand what is being told to you. Nobody is naturally doing that all the time, it's a state of mind that you have to intentionally activate, and it can't really be forced onto you.

All of the participants chose to engage in a conversation that was designed to convince them, which means they had all already accepted the possiblity that they are wrong a valid outcome. That's not really a state of mind you can trigger with a TV ad.

So, the warning that this could be weaponized isn't convincing to me.

If I found myself in a conversation with a person or chatbot who was clearly trying to convince me to flip a strongly held belief, like a major axis political affiliation, I would simply walk away because that's not a conversation I am willing to participate in.

So I don't think this is really weaponizable. Which actually means it's probably a good finding for society.

The study found that the models could convince anyone of anything, as long as it was allowed to cite facts. It could convince people of false things, but it needed to invent false facts in order to do so.

So as long as we keep training AI models to value facts and quality research (skills and values that are essential to be able to sell them as agentic workers), their influence on the opinions of society will tend to pull people away from beliefs that are unsupportable by facts. In the moments when those people are willing to accept a change in opinion, and they talk to a chatbot with doubts in their mind, even chatbots with no morals like Grok will tend to pull them away from conspiracy theories and similar ideologies and towards beliefs that are grounded in reality.

Some probably kinda strong counterevidence to that though is that this doesn't happen on all sites, only Amazon. I've never been prompted to make a new passkey after passkey login on the 12 other sites that I have passkeys in bitwarden for.

"No AI PR Descriptions" is a great rule because it does 3 things:

1. It's a hard binary that's easy to recognize and enforce

2. It establishes personal ownership for the submitter, making it psychologically difficult to submit something you don't understand. If a person has to describe what a change does, how, and why, then they must have looked at it and made a real attempt to understand it, because you can't map a territory you have never seen. This alone prevents "claude-code run amok" scenarios where devs have completely abdicated responsibility that are a real pain to clean up.

3. It informs the reviewer of the change's intent. A human PR tells reviewers what a thing is intended to do, as well as the authors belief about what it does, which they can use as a framework to critique the actual substance. It becomes easier to notice missed edge cases, code behavior that subtly differs from intent, or changes that would block off or complicate an intended future project direction or reduce maintainability.

And as a bonus, at time of writing, AI pr descriptions basically always suck. They are usually full of irrelevant implementation details, misplaced emphasis, weird presuppositions that seem to come out of nowhere and are really annoying to read, etc. That won't be true forever, they are always getting better, but it is true now, so that's a temporary 4th benefit.

Manuals, laws, and dictionaries are not designed to be read, but they are designed to be interpreted. In fact interpretability is what they maximize for. And interpretability isn't an absolute value, it is relative to an audience, this is why household equipment often comes with separate owners, installers, and service manuals, and why Webster publishes Elementary, Collegiate, International, and Mass-Market versions of its Dictionary.

Documents that are "meant for human consumption" in the way the top comment intended generally don't optimize for interpretability like code, manuals, and dictionaries do. Novels, letters, emails, internet comments, blogs, articles, nonfiction books, etc often forgo interpretaiblity for practical, expressive, or rhetorical effect. The best moments of these is even when they are confusing on purpose, because imprecise language is a gateway to deeper levels of meaning through fourth-wall breaks, metaphor, analogy, and humor. (all things that LLMs are notoriously bad at and don't seem to be improving on over the years)

Those other types of writing are intended to be read in full, every word, end-to-end, in order to educate, entertain, inform, or influence the reader who, and i think this is the critical part, does not drive the experience.

When I'm reading a news article, and you realize that you're reading the voice of chatgpt, I'm annoyed because it means that i've been strung along by a stupid machine that somebody else handed me off to to keep me occupied.

When I'm reading code, and I notice that it seems AI written for whatever reason, I might then choose to apply more scrutiny, maybe I try to suss out what the prompter intended to happen, so that I can validate whether that actually happened and if it was accomplished in an optimal way. But it's not insulting in the same way, because I'm still in the driver's seat of the experience, deciding on my own which parts to read and which parts to ignore as irrelevant to my investigation, because the only thing I want to get out of it an understanding of a part of the mechanical system that the code creates.

I'm guessing you didn't read what I wrote, or maybe I flubbed the communication in some ways because I was trying to be cooperative and complimentary towards you.

Edit: ok, you might object to me saying you're gullible, but the point I'm making is that that's not a bad thing and is actually evidence in favor of intelligence, the opposite of what the other guy was saying.

I'm gullible too. I'm also short, nearsighted, bad at communication, unable to bench over 90 lbs, and tend to spend way too much time researching fields irrelevant to my own work. Susceptibility to over-confidently incorrect messaging is a scientifically proven achilles heel of smart people. It's important for us to know that so that we can metacognitively account for it and hopefully avoid falling victim to it by way of scams, cults, and hallucinating AIs.

You mean you don't independently know one way or the other?

So you are taking it on his authority?

Because it was stated with confidence?

According to him, you must be one of those low IQ people.

Does that help to see the irony here, and why the original claim is not only clearly wrong but inherently elitist and insulting towards its own intended audience?

The alternative, the world that all intelligent people must live in at all times accordinng to the comment, is epistemic solipsism, that everything is presumed false, everyone is lying or erroneous, because there is no guaranteed way a priori to separate true statements from false ones.

> I doubt anyone is doing the research on it.

Not only is there research on it, this is like Psych/Journalism/Philosophy/Law/Communication 101 stuff. There are textbooks on it written over a century ago. It is the topic that the field of "Social Psychology" was invented to study.

The dominant theory is the Elaboration Likelihood Model, which describes and predicts based on situations when a person is more likely to doubt a claimed statement of fact, taking into account past experience with the source, cultural norms, motivation, time, concentration/nondistraction, and domain knowledge.

There is also tons of research specifically on the impact of intelligence on susceptability to persuasion by authoritative tone, and the reality is the *opposite* of what was claimed: smarter people who enjoy deep thinking are more likely to believe falsehoods (Price & Stone, 2004), because they perceive the overconfidence as exciting new information to think about, just like you are doing now.

So congrats, by uncritically believing the guy who incorrectly said you were low IQ for believing him, you actually provided evidence that suggests you are high IQ, because science shows that gullibleness is a trait of high-IQ people.

You could also be right-wing authoritarian, that's the other thing that strongly predicts gullibleness but I'm gonna guess you're smart instead.

What kind of compromise are we talking about? Was the device stolen? Then yeah, you need to rotate passkeys (and all your passwords, and remotely cancel all your live sessions, none of which is new).

Was the device hacked from afar and the data read off of it? The passkey is probably fine. You can rotate it if you want to, it's not a bad idea, I probably would to be certain, but you're not pwnd even if a bad guy got a shell for a while, heck even if they got a root shell.

I understood the question to be "should passkeys be rotated regularly" to which the answer is probably no because there's not really a compromise mechanism that periodic rotation defends against as far as I know.

Rotation defends against secrets that leak through normal use and without your knowledge, like a password entered into a phishing page or snooped over your shoulder or cracked from a breached hash. Passkeys aren't vulnerable to those things because they never travel over a network, are never seen by the server, are never seen by their own users.

You change the locks on your house when a key goes missing, or is known to be in the hands of somebody you don't want getting in, it's not something you do every three months just in case. Same with passkeys.

It's totally possible there's something specific about my situation, or the way it was set up in the first place that enables this, idk, but somebody else replied saying they have the same experience so it's not just me.

And even if it was just me, it's still clearly something wrong on the provider's implementation, because it should not be possible for software to sidetrack the user into a passkey enrollment flow, when that user logged in with a passkey to open the current session.

The remark is risible because it is an example of why it it false. Everyone is subject to influence by words, that's what words are for. Being "confidently incorrect" is also called being "mistaken", and everyone does it every day, including the commenter by the very act of saying it never happens.

It only takes a small modicum of thought to prove that the thesis is obviously completely false, yet it is stated with absolute sneering confidence and contempt.

By the commenter's logic, only someone with "low IQ" would believe their own words.

That whole episode caused the whole industry to shift away from RLHF, and towards RLAIF, RLVR, and DPO, and add a lot more safeguards, tests, and reward functions that push models in the direction of doing the opposite of what people want and confronting and strongly correcting their users, if it has determined the user is wrong.

When you review code, you're not taking it all in end to end like an email or memo or blog, you're looking for important symbols, recognizing structures and patterns, jumping between different files at different levels of reference and abstraction, and building a model in your head of the principles that make it work and their consequences for the overall procedure.

Code is "meant to be read" in the same way as a law, blueprint, policy document, or manual is, ie: jumping between relevant sections to get small pieces of information relevant to some task. The act of "reading" never starts at the beginning, it starts with a ctrl-f or flipping through an index or table of contents for a specific relevant symbol that is the only reason you are looking at the document.

Code is invention, not art (and should be covered by patent law, not copyright law, I will die on this hill)

What they aren't is moral, because of the orthogonality principle: you can't use facts and logic to discover correct moral beliefs. Morality is about values, goals, and the definition of "good". They must be provided to the robot by its creator, and those are things that are very hard to precisely describe in a way that is fully consistent with the speaker's intent in all possible scenarios, and agreeable by all other people.

Amazon prompts me to create a passkey everytime I log in, even when I logged in with a passkey, because my passkeys live in Bitwarden rather than my OS or browser.

And the confusing mechanism hurts there too: I'm always a little bit afraid that i'm somehow more in danger because I keep them in a vault that's shared on all my devices rather than a TPM, because whenever the protocol is explained the "it can't leave your device" part is highlighted as the main source of the security, except.... mine obviously do leave my device, with the vault, so.....

The purpose of rotating passwords is to cycle out potentially compromised ones, due to phishing attacks, keyloggers, shoulder snoops, etc. But those cannot exist with a passkey.