https://x.com/lossfunk/status/2034637505916792886

"After the paper was finalized, we ran agentic systems that mimic how humans would learn to solve problems in esoteric languages. We supplied our agents with a custom harness + tools on the same benchmark. They absolutely crushed the benchmark. Stay tuned"

A little harness engineering was enough!

- have bold, strong beliefs about how ai is going to evolve

- implicitly assume it's practically guaranteed

- discussions start with this baseline now

About slow take off, fast take off, agi, job loss, curing cancer... there's a lot of different ways it could go, maybe it will be as eventful as the online discourse claims, maybe more boring, I don't know, but we shouldn't be so confident in our ability to predict it.

While we're here, another thing that's annoying: the token counter. While claude is working, it read some files, makes an edit, let's say token counter is at 2k tokens, I accept the edit, now it starts counting very fast from 0 to 2k and then shows normal inference speed changes to 2.1k, 2.3k etc. So wanted to confirm: is that just some UI decision and not actually using 2k tokens again? If so, it would be nice to have it off, just continue counting where you left off.

Another thing: is it possible to turn off the words like finagling and similar (I can't remember the spelling of any of them) ?

Click on 'Play moves' to watch a replay.

I initially planned to run a chess tournament for LLMs but they are not good: besides obvious mistakes, they output incorrect moves, get stuck in loops by repeating the same moves and the smaller models fail to output valid json frequently. I thought the reasoning models like o3 mini might be good, but they are an incremental improvement in chess.

Feedback and suggestions for other games to explore welcome.

Comments URL: https://news.ycombinator.com/item?id=43076424

Points: 2

# Comments: 1

Judging by the quality of comments here and in the linked submissions, it's a good thing.

Also how well do these models work to extract structured output? Eg- perform ocr on some hand written text with math, convert to html and format formulas correctly etc. Single shot prompting doesn't work well with such problems but splitting the steps into consecutive api calls works well.

I know I can access the claude api through 3rd party sites + official partners, but there's no incentive to go through the trouble when llama 3 and gpt-4 apis work great for my use cases.

[0] https://support.anthropic.com/en/articles/8987200-can-i-use-...

Looks like level 3 ddos protection is only available on the enterprise plan, it's not included in the unmetered ddos protection.

I wonder if they use gpt-4 itself to generate the data to keep it upto date.

Slightly OT: everyone feels hiring is broken, can you list some things that are annoying from the employee and employer perspective? Here are some points:

- the process often stretches out over weeks and often months

- job posts often get 100s of applications, a lot are low effort applications, it just muddies the water for both sides

- ATS systems/job boards are annoying with the need to create an account on many sites, some forms have more than 20 questions, often asking what's already there in the resume.

A question to everyone: What would a good application process look like? For me, it should just solve the above mentioned problems. I send an email with my resume, a few sentences about why I might be a good fit for the role/what interests me about the company. The jobs@.com email address could be linked to some Saas product which makes it easy for the employer to go through the applications and further communication about video calls or take home assignments or whatever are all in this email thread. The employer can set the stage of the application such as 2/5 or whatever, they can mark it as rejected or accepted after all rounds to trigger automated emails etc. Is there any Saas like this? (I can build this in a week if it doesn't exist, but no clue how to market it/get users, any pointers in case me or someone else builds this?)

[0] https://openai.com/blog/function-calling-and-other-api-updat...

I've been working on this on and off for a few weeks now. The idea is to let LLMs interact with the outside world using function calling. Eg- I built a simple version of chatgpt web browsing using function calling. There's a django site which is responsible for the auth, managing chats etc and then there's a flask site that's responsible for running functions in a gvisor sandbox. Users will be able to upload their functions as docker images and I'll run them in a sandbox.

There is a lot of self doubt at this point: who is this for? Will people pay for usage? Should I open source the entire thing or part of it? To add to this: I have no online audience, how do I promote this?

I do have some answers to these questions. Eg- should I open source the flask site and get llama 2 to use it? It provides value to the community + hopefully I can get some attention for the project.

I would appreciate any feedback on this project. Please send me an email if you want some openai credits added to your account.

Edit: use testaccount/demouser to login, I've added some credits for testing.

More of a meta comment: this is pretty much why I don't install any extensions in my browser except an ad blocker.

You can use this as an opportunity to teach your friend about security so it doesn't happen again.

Would you be willing to give a few more details about what happened? I'm not interested in the identities of the companies or people, just interested in a high level overview of what happened. We don't hear these stories often.