"At some point we’d get someone to run the team, but he/she probably shouldn’t be on the governance board"

"generally, safety should be a first-class requirement"

"Probably better to have a standard C corp with a parallel nonprofit"

"Because we don't have any financial obligations, we can focus on the maximal positive human impact"

"The underlying philosophy of our company [OpenAI] is to disseminate AI technology as broadly as possible as an extension of all individual human wills, ensuring, in the spirit of liberty, that the power of digital intelligence is not overly concentrated and evolves toward the future desired by the sum of humanity"

"The outcome of this venture is uncertain and the pay is low compared to what others will offer, but we believe the goal and the structure are right"

"do you have any objection to me proactively increasing everyone's comp by 100-200k per year?"

"The output of any company is the vector sum of the people within it."

"it's totally OK to not share the science (even though sharing everything is definitely the right strategy in the short and possibly medium term for recruitment purposes)"

"Frankly, what surprises me is that the AI community is taking this long to figure out concepts. It doesn't sound super hard."

"Powerful ideas are produced by top people. Massive clusters help, and are very worth getting, but they play a less important role."

"Deepmind is causing me extreme mental stress."

"At any given time, we will take the action that is likely to most strongly benefit the world."

"Would be worth way more than $50M not to seem like Microsoft's marketing bitch."

"Ok. Let's figure out the least expensive way to ensure compute power is not a constraint..."

"Within the next three years, robotics should be completely solved . . . In as little as four years, each overnight experiment will feasibly use so much compute capacity that there’s an actual chance of waking up to AGI"

"We think the path must be: AI research non-profit (through end of 2017), AI research + hardware for-profit (starting 2018), Government project (when: ??)"

"Satisfying this means a situation where, regardless of what happens to the three of them, it's guaranteed that power over the company is distributed after the 2-3 year initial period"

"As mentioned, my experience with boards (assuming they consist of good, smart people) is that they are rational and reasonable. There is basically never a real hardcore battle. . ."

"The current structure provides you with a path where you end up with unilateral absolute control over the AGI. You stated that you don't want to control the final AGI, but during this negotiation, you've shown to us that absolute control is extremely important to you. As an example, you said that you needed to be CEO of the new company so that everyone will know that you are the one who is in charge. . ."

"Specifically, the concern is that Tesla has a duty to shareholders to maximize shareholder return, which is not aligned with OpenAI's mission"

"During this negotiation, we realized that we have allowed the idea of financial return 2-3 years down the line to drive our decisions . . . this attitude is wrong"

"i remain enthusiastic about the non-profit structure!"

". . .apparently in the last day almost everyone has been told that the for-profit structure is not happening and he [Sam] is happy about this"

"Our goal and mission are fundamentally correct"

"We also have identified a small but finite number of limitations in today's deep learning which are barriers to learning from human levels of experience. And we believe we uniquely are on trajectory to solving safety (at least in broad strokes) in the next three years."

"Our biggest tool is the moral high ground. To retain this, we must: Try our best to remain a non-profit. AI is going to shake up the fabric of society, and our fiduciary duty should be to humanity. Put increasing effort into the safety/control problem, rather than the fig leaf you've noted in other institutions. It doesn't matter who wins if everyone dies. Related to this, we need to communicate a "better red than dead" outlook — we're trying to build safe AGI, and we're not willing to destroy the world in a down-to-the-wire race to do so."

"The sharp rise in Dota bot performance is apparently causing people internally to worry that the timeline to AGI is sooner than they’d thought before."

"This needs billions per year immediately or forget it."

"all investors are clear that they should never expect a profit"

"We saw no alternative to a structure change given the amount of capital we needed and still to preserve a way to 'give the AGI to humanity' other than the capped profit thing, which also lets the board cancel all equity if needed for safety. Fwiw I personally have no equity and never have."

I think this technique is going to be valuable for controlling the output distribution, but I don't find their "introspection" framing helpful to understanding.

https://sites.math.rutgers.edu/~zeilberg/OPINIONS.html

Word generating machine will not "figure out how to cure cancer" but it could help, obviously. AI is extremely valuable tool but it does not work on my behalf, except in same sense a coin sorter does. It's a tool. I still see this thing confuse left and right (no exaggeration), which would be fine - tools aren't perfect - except for all bullshit from VCs. That is where the danger lies, not with the tool but idolatry.

I am concerned the system encourages suicide, delusional thinking etc. They need to work that out immediately. Must be held to at least the standard of a lawn mower or couch with regard to safety. Probably should make it safe before hooking it up to 10GW power plant. Does not help perception that author of this blog saying how awesome future will be is also building a hideaway bunker for himself.

This is all on you at OpenAI, Anthropic, Microsoft, Twitter etc. Whatever happens.