I just wish we had a way to actually benchmark them properly though. Still seems no one has solved the problem of software architecture, brittleness and bloat as the codebase grows. Models love to add stuff, but they rarely clean up as they go. In a perfect world they'd do both near equally as they're developing.

It would be nice if there was an "architecture quality" benchmark that distilled the essence of what it means to have a good architecture, but I suppose that's an open research question with a lot of variables? Like how is good architecture actually quantified and measured? Is there a mechanism that can be re-used across all codebases to clearly denote one that is good and one that is bad, or is it highly subjective and depend on the lens you're looking at it from? Is there a lot more to it than just "how much refactoring effort is required to extend this in the future?".

Surely this is something that has been well researched - yet I never really hear anything about it. Makes me wonder why.

Their license terms are also incredibly generous and allow commercial use, modification, etc, at no cost.

It highlights the difference between companies like Nvidia and Anthropic to me, where one is clearly all about the money and power, and the other is doing it because they genuinely want to accelerate progress and make cool stuff as the driving factor. It's no surprise therefore, that Nvidia is the worlds largest open-source contributor to AI, with over 800 open-weight models.

Of course, these models run on Nvidia hardware, so they benefit from it as a company. But with that healthy mindset, they found a way to contribute that not only benefits everyone, but also benefits themselves.

Contrast to Anthropic, who has gone the complete opposite direction. Closed off everything, restricting everything, fearmongering progress, regulatory capture attempts, the list goes on. I mean, they won't even agree on using AGENTS.md as a standard because CLAUDE.md is free marketing for them. That's the level of disgusting greed we are dealing with...

From a game theory perspective, the cooperative strategies tend to win. As a result, Nvidia has set themselves up for a lifetime. Anthropic however, is playing a strategy of winner takes all, and they're happy to see the world and the entire AI industry collapse in the process.

It's the most logical solution for AI anyway, considering that it's training on humanities collective knowledge. It should be more of a public-funded and public-access resource, rather than something greedy tech companies distribute like crumbs while they use unlocked powers internally to clone all of our businesses and swallow the economy.

Right now, and likely forever, because biological threats can be sanctioned at a supply-chain level, the risk of AI is all digital. Fraud, phishing scams, spam, hacks, etc.

The only way we harden the worlds infrastructure to the point that it can withstand attack from bad AI is if we have an abundance of access to frontier intelligence to develop countermeasures.

Otherwise, bad actors will develop these capabilities behind closed doors and use them to hold the world hostage and cause irreparable harm. There's no putting the genie back in the bottle. Good and open-access AI and the people using it are the digital immune system.

If there's an asymmetry where bleeding edge is gated off to only a small group, and allowed to gain exponential power over the immune systems defense grid, the slightest infection will lead to death of the host.

Bad AI is only countered by having a majority of good, open-access and open-source AI to keep it in check, where the good AI can overpower the bad. The moment you destroy that balance is the moment a bad actor gains exponential advantage and the ability to hold the whole world hostage forever.

Step 1: "OMG, the AI hacked a researcher eating a sandwhich in the park!"

Step 2: Journalists use that great clickbait to generate profit, which generates publicity for Anthropic

Step 3: Rinse repeat

If the threat of LLMs was treated relative to the actual capabilities of them, and we weren't all being lied to by Anthropic and their army of millions of social media bots and backing media companies and mouthpieces, we'd be going in a much healthier direction. Working out the kinks/supply chain risks and developing sound, long-term countermeasures to the ACTUAL risks.

The only threat to the world is if progress is not open-sourced, democratized and in lockstep with capability. The moment it becomes a scenario of: Only a small group get access to frontier intelligence, is when it gives that small group power over everyone else in the world, and wildly increases the risk of a nuclear level event that WILL be exploited eventually - as the divide between the haves and the have nots accelerates in an exponential fashion. Bad AI is countered with an abundance of good AI that has been used to stay ahead of bad AI. The moment your bad AI outpowers the army of good AI it is game over for humanity. The strength of open-source and open-access AI is the difference between humanities permanent enslavement or extinction versus a prosperous future.

It doesn't help that most of the employees at Anthropic have willingly sold their souls out of short term greed and gaslight themselves into thinking that they're actually doing the right thing to justify their own greed to themselves, while building up an echo-chamber and culture of feel good lies within the company so they can sleep at night, and pat each other on the back. They go along with this because they get paid massive chunks of money from Anthropic, and their shares will be worth more money if Anthropic can swallow the worlds economy at the expense and enslavement of everyone else. What good is that money when you have to sell out humanity in the progress though. You, at Anthropic, is that the legacy you want to leave?

People need to start calling this out before it's far too late. If you work at Anthropic - time to start talking to your colleagues in an honest manner.

Unfortunately though, they're not smart enough to realize the long term damage to the industry that they're doing without any hint of remorse will negatively affect them and will have the opposite effect: Highlighting how imperative it is we all switch to open-source and remove our reliance entirely from them.

So not only are they going to take the whole industry down out of their own greed and stupidity and ruin it for everyone in the short term, but they're going to put themselves and the other labs out of business in the long term. Well done Anthropic. Well done Dario. You played yourself. 5d chess.

The funny thing is that the model isn't even impressive. I'd still use ChatGPT over it for anything other than design work. As soon as OpenAI cracks design, I'll never touch an Anthropic model again.