Since it's a competitive setup, and there are hundreds of board games that could be implemented, this arena approach shouldn't become instantly saturated like other benchmarks, although it's certainly possible for individual labs to finetune their models for the specific games selected.

A notable gap is the exclusion of o1 and Google's Gemini's 2.5. I may add o1 if there's enough interest, but the arena is a bit expensive to pay for out of pocket, and Gemini's rate limits were too low for me to add it right now.

Comments URL: https://news.ycombinator.com/item?id=43556853

Points: 2

# Comments: 0

It could be that the moral authority stems from having as much of a full picture as a single person can have over the entire lifecycle of the company, but I think a lot is also just the effect of "I got you here."

I'm glad pg named this effect, since I've talked about the related phenomenon for CTOs with many people.

Perhaps it doesn't matter to the health of your network, but if it leads to a customer's account being disabled due to incorrectly assigned abuse, surely it would matter to them.

The prompt itself can trigger the features, so if you say "Try to weave in mentions of San Francisco" the San Francisco feature will be more activated in the response. But having a global equalizer could reduce drift as the conversation continued, perhaps?

> Any kind of content (art/code/sound/etc) created with the help of AI tools during development.

But that none of the categories the author of this post identified included code, only visual, audio and text content.

> The single number that should summarize your expectations about any LLM is the number of total flops that went into its training.

One thing I've been curious about is whether a model that's trained well beyond the Chinchilla level of compute will suffer more from quantization. All of that information has to live somewhere within the weights, so it stands to reason that you may have to keep more bits of information to keep that performance benefit.

If so, it would also mean that a smaller model that's been "overtrained," but which can't be quantized without suffering quality loss isn't necessarily cheaper for inference than a larger model which isn't overtrained, but which can be aggressively quantized. I haven't seen anyone discuss this, but maybe there's a paper on it.

If you could characterize what level of overtraining leads to quality loss at different levels of quantization, you could possibly figure out a more optimal model for overtraining. E.g. if you train with 10T tokens and you see quality loss at 4 bit, and you train with 20T tokens and see quality loss at 6 bit, you can fit a curve to those data points to estimate the maximum amount of tokens the model can train on with the current methodology.

Yeah, I'm pretty sure that would be physically impossible, unless they are planning on hosting most of the conference on the grounds, parking lot and maybe surrounding streets.

https://palaceoffinearts.com/tour/

When I learned it was written in the 1980s, I wasn't exactly shocked. But then, I learned it was written by the Klein bottle guy, and that really was shocking. It's become one of my favorite books.

Even if it turned out to be practical to double the volume of cargo you could carry, it seems unlikely that it would allow you to double the weight of the cargo, since the engines and the airframe have all been designed around the same set of engineering requirements. The best case scenario would be a decrease in the cost of volumetric cargo, with dense cargo staying the same.