Very true.

In my day to day as a Product Manager working in a team that ships AI products, I often found myself wanting to do 'quick and dirty' LLM-based evaluation on conversation transcripts and traces. I didn't need anything fancy, just 'did the agent answer the question', 'did the agent cover the 5 things it needed to' - that type of thing.

I found myself blocked by 'Gemini in Google Sheets', it was too slow and cumbersome, and it didn't handle eval changes well - particularly when trying to associate evals with ground truth. And because I was exploring or working on new and experimental features, it wasn't helpful to try and set up something more robust with the team.

To fix the problem I eventually learned to call the OpenAI API in Python, but I really felt that I wanted a 'product' to help me and potentially help others who need answers fast - outside of building infrastructure and pipelines.

So over the last few weeks I built: https://beval.space

It has: - LLM-as-judge evals: boolean checks (yes/no), scores (1-5), categories, and freeform comments - Reusable eval definitions you can run across different datasets - Ground truth labelling so you can compare eval versions against human judgments - Per-trace reasoning so you can see why the judge scored something the way it did - An example dataset so you can try it without having your own traces ready

One of our early users described it as 'quick n dirty evals when you don't want to touch a shit load of infra.' I'm trying to figure out if that's a common need or just a niche thing.

Free during beta. Would love HN's take — what's missing, and would you actually use something like this?

Comments URL: https://news.ycombinator.com/item?id=47566753

Points: 2

# Comments: 1

I feel like PMs coding unlocks a whole new category of work, mainly addressing the long tail of cool ideas/small optimisations that ordinarily would not be addressed. Time will tell how valuable these items are in the long term.

And I say this as a PM.

In my day to day as a Product Manager working in a team that ships AI products, I often found myself wanting to do 'quick and dirty' LLM based evaluation on conversation transcripts and traces. I found myself blocked by 'Gemini in Google Sheets', it was too slow and cumbersome, and it didn't handle eval changes well. And because I was exploring, it wasn't helpful to try and set up something more robust with the team.

To fix the problem I eventually learned to call the OpenAI API in python, but I really felt that I wanted a 'product' to help me and potentially help others.

So this weekend I built https://beval.space

Personally I find this kind of approach infantilizing. I don't think it's fair to have an expectation of yourself or of your colleagues to spell out in Google Calendar what you/they are spending their time doing throughout the day. People should look at gaps in people's diaries not as opportunities for new meetings, but as time someone is probably spending doing something important or interesting.

Classify all the tasks with an impact / ease score from 0-5. Multiply the numbers together to understand the relative priority of each item.

It won't be perfect but it will a good place to start.

You also do this at a product level instead, and invest all your time onto one product to begin with. This might reduce your switching costs working between projects/products.

For now, I’m just sticking to using it for daily notes, but I feel there’s so much I’m missing.

UX feedback: When I scroll to the bottom of the page linked in the post, and click on the API/Pricing links in the footer, it fails to redirect me.

Comments URL: https://news.ycombinator.com/item?id=43567781

Points: 1

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=42986335

Points: 66

# Comments: 45

Comments URL: https://news.ycombinator.com/item?id=42763214

Points: 2

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=42549147

Points: 132

# Comments: 75

Comments URL: https://news.ycombinator.com/item?id=42422337

Points: 3

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=41922827

Points: 7

# Comments: 6

Comments URL: https://news.ycombinator.com/item?id=41862726

Points: 1

# Comments: 0