Also, model providers are not interested to have their models compared head-to-head under identical conditions. And “Model A is better than Model B” is almost meaningless by itself. Better for what task? With what prompt? What inputs? What budget? What failure tolerance?

It would be nice to have a place where users could run their own benchmarks, define evaluation criteria for their actual use cases, and make those runs verifiable by others.

In the end, it became heavy overengineering: people no longer understood not only the repo itself, but also the extra layer describing it. Meanwhile, coding assistants are already quite good at reading codebases directly.