https://developers.openai.com/api/reference/resources/respon...

I don't know their internal eval, but I think I have heard it does not hurt or improve performance. But at least this parameter may affect how many comments are in the code.

I guess they can be a contributor there.

* Likely they will seek regulation that would ban some models. Not sure this can work, but they will certainly try.

* Likely they will not release some of their next models in the API.

Anthropic seems the best in this. Everything is in the API on day one. OpenAI tend to want to ask you for subscription, but the API gets there a week or a few later. Now, Gemini 3 is not for production use and this is already the previous iteration. So, does Google even intent to release this model?

I am interesting if I can run it on a 24GB RTX 4090.

Also, would vllm be a good option?

What you need to do is to match the distribution of how the models were RL-ed. So you are right to say that "do X in 200 lines" is a very small part of the job to be done.

However, going above 75%, it is likely about the same. The remaining instances are likely underspecified despite the effort of the authors that made the benchmark "verified". From what I have seen, these are often cases where the problem statement says implement X for Y, but the agent has to simply guess whether to implement the same for other case Y' - which leads to losing or winning an instance.

I would have expected good providers like Together, Fireworks, etc support it, but I can't find it, except if I run vllm myself on self-hosted instances.

Somehow if I take the best models and agents, most hard coding benchmarks are at below 50% and even swe bench verified is like at 75 maybe 80%. Not 95. Assuming agents just solve most problems is incorrect, despite it being really good at first prototypes.

Also in my experience agents are great to a point and then fall off a cliff. Not gradually. Just the type of errors you get past one point is so diverse, one cannot even explain it.

For some of these I see something like 15k followers on X, but then no LinkedIn page for example. Website is always a company you cannot contact and they do everything.

I ended up disabling copilot. The reason is that the completions do not always integrate with the rest of the code, in particular with non-matching brackets. Often it just repeats some other part of the code. I had much fewer cases of this with Cody. But, arguably, the difference is not huge. But then add on top of this choice of models.

I wonder if US monopoly regulation actually starts to work well, which I see some signs of happening, will all this license revert back to fully open source?

Yes, some languages are underrepresented and there are some thresholds. But exactly, it is well known that putting the threshold just slightly above or below will probably not materially affect the model.

But also as users, some were expecting the M series are so good that they are going to take many markets by storm. And it seems it is not happening.

But here is what is visible:

The M2 core is probably in the same ballpark as Zen 4 core, likely a tiny bit below. That may become very tiny if Zen 4 core runs at lower frequency to equalize the power. This doesn't account for the AVX512 of Zen4.

24 M2 cores manage to beat 16 Zen 4 cores also at lower power, but these are different products. Zen 4 does scale to far more cores, 96 in an EPYC chip. AMD and Intel have far more investments in interconnects and multi-die chips to do these things.

The M2 GPU is in the same league as a 300$ mid-range nVidia card. It is not competitive at all - Apple produces the largest chip it can manufacture to go against a high margin smaller chip that nVidia orders.

Again all of this doesn't mean each product is not good on its own.