Comments URL: https://news.ycombinator.com/item?id=49779979

Points: 3

# Comments: 0

I'm more than capable of training a bert classifier in fact in 2019 I had trained many custom berts and was running them on hundreds of millions of documents a day.

I don't want to manage GPUs / CPUs now. I don't want to maintain my corpus and retrain as my product's data distribution shifts. The list of things I don't want to do goes on and on and on. And I'm happy for them to be someone else's problem.

I do just want a reasonably good general classifier served to me with a great devex and calibrated confidence scores to help me figure out when to fallback to another model.

This is very obviously not the type of situation and behavior marginalia_nu and then I were commenting about.

But ARE you doing anything about it? Because unless you're actually taking action it's basically just self flagellation to the benefit of no one. Like if you have some measure of power (community action, organized resistance, political lobbying) and are taking action that's great, reading articles that make you sad and angry and then going to bed isn't doing anything.

I think you have far more leverage than you think in most places still.

I often see this repeated, and it is not true task to task. I work on this daily and we have several tasks where long context is advantageous and our evals against a whole battery of models with different windows show it as being so.

This is why having good evals for the tasks you're working on is so important.

I do grant it's a good rule of thumb.

There are many applications that will benefit from the strength in audio here and until z.ai and co work in visual this could be very strong for general agentic applications, though I see there's a bit of weakness in the benches for areas that might make that less true.

Like all models need to slap it in your harness and do proper evals on the tasks you care about.

Config -> Switch models when a message is flagged -> false

That should stop it entirely when a message is flagged and then you can come back without Opus having potentially made a mess.

METR already redid the study at a later date and now finds a likely 18% speedup

"For the subset of the original developers who participated in the later study, we now estimate a speedup of -18% with a confidence interval between -38% and +9%" (note their use of - and + here could be slightly confusing but they do mean 18% faster per the post)

https://metr.org/blog/2026-02-24-uplift-update/

FAANG doesn't straightforwardly retain the best technical talent, and the median task there is routine.

Lol. I don't know I was talking with a guy yesterday who left a FAANG clearing 1.4 million / yr who's now running his own successful startup. My sample is successful founders backed by top tier VCs or exited founders who have done much better than FAANG. If you can't do more you stick around the FAANG, those who can go and do it.

Fwiw we both agree that LLMs should not design systems. I do the design, but otherwise I don't get how this is true, the success of a design is indicated by long term success in the system it built. You can measure this against success in the task it was deployed for via performance metrics for one. And then from a developer standpoint how easy it was to maintain later on. Success of a system is a measurement over time, but it's not some quality that can only be measured by those who built it.

> Off-topic but having worked in other companies as well, I can guarantee you that this is not the case. The skill of engineers in FAANGs and other "top tier" companies is much higher than average.

I have first hand knowledge of this so I agree to disagree. Being surrounded by google, aws, and meta folks my understanding is the best people leave faang when they get the itch to do something better with their time.

https://x.com/yaroslavvb/status/2067367657272422584 https://x.com/voratiq/status/2067667800643268928 https://arena.ai/leaderboard/agent

Edit: Mis remembered the timeline I saw not 2 weeks, 3 months, but still I think my point stands.

Every time I see an anecdote like this, A it reaffirms my belief that FAANG devs are fairly mediocre on the whole (not saying this is you, obviously there are good FAANG devs) and B it reaffirms my belief that the developers who kind of give up their thinking like this are really using the tool wrong or didn't really care about the work before AI either so its now just a quick means to an end.