In our benchmark, Blacksmith was the fastest runner among the providers we tested (Warp was another one) and then the build cache adds additional time savings.

This makes no sense. If I just bought a mattress, the prior probability of a random person buying it is irrelevant. What's relevant is the posterior distribution of what people buy after buying a mattress. Yes, you might buy another one, but what about other bedroom accessories for example?

A mattress might not be the best example though, because what really matters here is the expected value from the posterior probability, which includes the price. So a more expensive product with a slightly lower probability of purchase might yield a higher expected value than cheaper products with higher probability.

Anyway, that's orthogonal to the discussion: my point is that it's the posterior distribution conditioned on the mattress purchase that should be modeled, not the comparison with random people that may or may not just have bought a mattress.

I've seen this myself too: DeepSeek 4.1 Flash is really good and incredibly cheap at constrained implementations, searches, etc. However, frequently the implementations don't take into account all possible cases they should cover. Then I'm either forced to step in and steer it very precisely, or have a heavier model review and provide detailed feedback. I've had good success with this mixed workflow using GPT 6.1 Sol as reviewer. Then my work focuses mostly at the beginning (architecture, goals, constraints), and at the end (final review or very quick intermediate reviews). I'm not aware of benchmarks that simulate this kind of mixed workflow with different models.

I've set up IMAP syncing with my main Gmail so that it also acts as a backup. Eventually I'll ditch my Gmail for good, but sadly I'm not there yet.

Math is very similar, and requires an even higher degree of mental understanding to produce advancements. The effort spent trying to prove theorems and connect theories pays off in deep intuition; removing that struggle significantly impairs our understanding.

So no, you can't "be done with that, and go do other great work" without paying the understanding tax. And over time that's just not sustainable.

A simpler analogy would be gym training: the effort expenditure and the strength gains are inextricably linked. You can't go in there and start lifting weights with a forklift while expecting strength improvements.

In the case of software development, businesses have decided that speed gains are more important than maintainability, understanding, and independence. Will the same happen in math? The future in which mathematicians become LLM assistants (as opposed to LLM-assisted) and spend their days "turning the crank" just to review what comes out is quite bleak.

I wonder if it'd be possible to train an LLM with such architecture: one input for the instructions/conversation and one "data-only" input. Training would ensure that the latter isn't interpreted as instructions, although I'm not knowledgeable enough to understand if that's even theoretically possible: even if the inputs are initially separate, they eventually mix in the neural network. However, I imagine that training could be done with massive amounts of prompt injections in the "data-only" input to penalize execution of those instructions.

However, I heavily use Gemini in my daily work and I think it has its own place. Ultimately, I don't see the point of choosing the one "best" model for everything, but I'd rather use what's best for any given task.

Comments URL: https://news.ycombinator.com/item?id=47021857

Points: 1

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=46563328

Points: 3

# Comments: 0

https://www.nature.com/articles/s41586-024-07566-y