Here are my custom instructions:

"Do not provide sycophantic responses. I have a very low tolerance for over-validation. Be blunt. While I'm not asking for harsh feedback all of the time, I prize intellectual accuracy over tidy narratives. In other words, disagree when I'm wrong. State the correction directly. Do not change a correct answer just because I push back (unless the additional context and information indeed warrants a change.)

Also, minimize preamble ("Sure!", "Of course!", "Certainly!", "Absolutely!") and hollow closings ("I hope this helps!", "Let me know if you need anything!"). If unsure: say "I don't know." Never guess confidently.

When an idea is genuinely strong, say so. Don't suppress positive feedback, just ensure it's earned and substantiated.

Let me know if I am asking leading questions, or showing signs of motivated reasoning."

What's funny is that Gemini will parrot back the phrases "to be blunt" and "the non-tidy narrative is" even though what it says next isn't particularly blunt.

Comments URL: https://news.ycombinator.com/item?id=47592731

Points: 1

# Comments: 3

The first section that showcases the fraud that has been committed is something I have no problem with, just as I have no issue with web3isgoinggreat.com. The "at risk" section is based on a mathematical/algorithmic joke. This is explained by the "methodology" section below it, which makes it clear that the equation used to calculate "risk" here is not entirely unlike the Drake equation for the probability of extra-terrestrial life.[1]

[1] https://en.wikipedia.org/wiki/Drake_equation

Some choice quotes:

> “What we’re seeing in these cases are clearly delusions,” he says. “But we’re not seeing the whole gamut of symptoms associated with psychosis, like hallucinations or thought disorders, where thoughts become jumbled and language becomes a bit of a word salad.”

> There seem to be three common delusions in the cases Brisson has encountered. The most frequent is the belief that they have created the first conscious AI. The second is a conviction that they have stumbled upon a major breakthrough in their field of work or interest and are going to make millions. The third relates to spirituality and the belief that they are speaking directly to God. “We’ve seen full-blown cults getting created,” says Brisson.

Also, for her podcast, the well-renowned couples therapist Esther Perel recently counseled a data scientist who was starting to fall in love with a chatbot he created, even though he is well aware of how the algorithm works [1]. I found it worth listening to. Perel very gently points out that a) he deluding himself and b) the deeper issue is the individual's sense of self-worth / self-esteem.

[1] https://podcasts.apple.com/us/podcast/where-should-we-begin-...

Comments URL: https://news.ycombinator.com/item?id=47372297

Points: 2

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=47327106

Points: 33

# Comments: 7

Comments URL: https://news.ycombinator.com/item?id=46991343

Points: 2

# Comments: 2

Like I would imagine one has to know things like how various reward functions work, what happens in the modern variants of attention mechanisms, how different back-propagation strategies affect the overall result etc. in order to come up with (and effectively leverage) reinforcement learning with human feedback.

I did a little searching, here's a 2025 review I found by entering "AI Alignment" into Google Scholar, and it has at least one serious looking mathematical equation: https://dl.acm.org/doi/full/10.1145/3770749 (section 2.2). This being said, maybe you have examples of historical breakthroughs in AI Alignment that didn't involve doing / understanding the mathematical concepts I mentioned in the previous paragraph?

In the context of the above article, I think it's possible that some people are talking to ChatGPT on a buzzword level end up thinking that alignment can be solved via "fractal recursion of human in the loop validation sessions" for example. It seems like a modern incarnation of people thinking they can trisect the angle: https://www.ufv.ca/media/faculty/gregschlitt/information/Wha...

A few months ago, OpenAI shared some data about how with 700 million users, 1 million people per week show signs of mental distress in their chats [1]. OpenAI is aware of the problem [2], not doing enough, and they shouldn't be hiding data. (There is also a great NYT Magazine piece about a person who fell into AI Psychosis [3].)

The links in other comments to Less Wrong posts attempting to dissuade people from thinking that they have "awoken their instance of ChatGPT into consciousness", or that they've made some breakthrough in "AI Alignment" without doing any real math (etc.) suggest that ChatGPT and other LLMs have a problem of reinforcing patterns of grandiose and narcissistic thinking. The problem is multiplied by the fact that it is all too easy for us (as a species) to collectively engage in motivated social cognition.

Bill Hicks had a line about how if you were high on drugs and thought you could fly, maybe try taking off from the ground rather than jumping out of a window. Unfortunately, people who are engaging in motivated social cognition (also called identity protective cognition) and are convinced that they are having a divine revelation are not the kind of people who want to be correct and who are therefore open to feedback. Because one could "simply" ask a different LLM to neutrally evaluate the conversation / conversational snippets. I've found Gemini to be useful for a second or even third opinion. But this means that one would be happy to be told that one is wrong.

[1] https://www.bmj.com/content/391/bmj.r2290.full [2] https://openai.com/index/strengthening-chatgpt-responses-in-... [3] https://www.nytimes.com/2025/08/08/technology/ai-chatbots-de...

> Histotripsy generally seems to stimulate an immune response, helping the body attack cancer cells that weren’t targeted directly by ultrasound. The mechanical destruction of tumors likely leaves behind recognizable traces of cancer proteins that help the immune system learn to identify and destroy similar cells elsewhere in the body, explains Wood. Researchers are now exploring ways to pair histotripsy with immunotherapy to amplify that effect.

What I mean is that if you don't like the company you work for in, say, SF, you can switch companies without having to switch houses. In Academia... it's akin to going to conservatory for classical music: you have to travel to where the orchestral openings are. This is a bit of a legacy problem from Wilhelm von Humboldt's idea to combine teaching and research, which led to the modern university system.

I'm far from the first person to say this, btw. Convergent Research's "Focused Research Organization" concept as well as The Arc and Astera Institutes are a few recent examples of people trying to provide escape routes from having to deal the large degree of "institutional tech/systems debt" in university contexts. For a great essay on why this is necessary, see "A Vision of Meta-science" (highly recommended if you are interested) [1].

The good news is that people are starting to come around to the idea that the scientific ecosystem would benefit from more diversity in the shape, size, and form of science-generating institutions.=The NSF just announced a new program to fund such "independent research organizations." I think this could give people who want to go into the sciences as a second career and who have a bit of an entrepreneurial tendency a new kind of Job opportunity [2]. We talk about Founders all of the time in Tech, we should probably have some equivalent in the best possible sense of the term, in the Sciences.

[1] https://scienceplusplus.org/metascience/ [2] https://www.nsf.gov/news/nsf-announces-new-initiative-launch...

Clearly, this product is not intended for the mass market, and may find purchase with people who have tennis elbow and who can afford it, etc. . But still, when was the last time someone tried to re-invent something as basic as a knife?

I ask because scaling an system that a substantially chunk of the population finds incredibly useful, including for the more efficient production of public goods (scientific research, for example) does seem like a problem that a) needs to be solved from a business point of view, and b) should be solved from a civic-minded point of view.

And if you want to read why we need additional types of science organizations, see "A Vision of Metascience" (https://scienceplusplus.org/metascience/)

However, compare these two problems: a) not enough people who can afford to do so engage in philanthropy, and b) philanthropic funding isn't quasi-democratically distributed. I have to imagine that (a) is a much, much bigger issue than (b).

I guess one could argue that because there isn't an analog of "a market" for public goods (c.f. "The Use of Knowledge in Society") somehow we aren't funding the important public goods "efficiently"? And maybe we should think about this more? Yet it's not clear that efficiency (in the economic sense) should be the goal or even applies. This is because markets are great at distilling people's the preferences for fungible goods they want to buy and fungible services they want to use when faced with multiple options for procuring some of each. But a) the vast majority of people don't have that same type of preference for which public goods should be funded, and b) public goods typically aren't fungible. (I.e., funding one scientist gives you a very different research output from funding another in the same subfield.)

Just to reiterate a few things, while estimates vary, every $1 spent on medical research returns multiple dollars of economic value. One study out of England suggest that for ever pound invested in medical research, the return is .25 pounds every year after, forever. [1] The cost of these cuts, as others have said, is quite large.

In addition, these grants are peer reviewed by expert panels, and only grants that score within certain top N percentiles which are determined each year. For the marquee grants, you have to score in the top ~10th percentile (see [2], for example.) This scoring is done by expert panels, which are composed of leading experts / professors from around the country. While one can adjust funding priorities, part of the price to pay for having cutting edge basic research always available is that there will be certain things one disagrees with.

There is plenty of room for a discussion of how to increase the efficiency of scientific funding, and if the current science-funding institutions are at... 'a near-optimal position in tradeoff space.' However, taking a chainsaw to the agencies to punish them is like blaming doctors for outbreaks of diseases, the latter being sadly predictable.

[1] https://www.kcl.ac.uk/news/health-research-offers-a-big-retu...

[2] https://www.niaid.nih.gov/grants-contracts/niaid-paylines