Let's take the example of the GitHub pr slack bot from the blog post. I would expect 2-3 evals out of that.

Starting at the core, the first eval could be that, given a list of slack messages, it correctly identifies the PRs and calls the correct tool to look up the status of said PR. None of this has to be real and the tool doesn't have to be called, but we can write a test, much like a unit test, that confirms that the AI is responding correctly in that instance.

Next, we can setup another scenario for the AI using effectively mocked history that shows what happens when the AI finds slack messages with open PRs, slack messages with merged PRs and no PR links and determine again, does the AI try to add the correct reaction given our expectations.

These are both deterministic or code-based evals that you could use to iterate on your solutions.

The use for an LLM-as-a-Judge eval is more nuanced and usually there to measure subjective results. Things like: did the LLM make assumptions not present in the context window (hallucinate) or did it respond with something completely out of context? These should be simple yes or no questions that would be easy for a human but hard to code up a deterministic test case.

Once you have your evals defined, you can begin running these with some regularity and you're to a point where you can iterate on your prompts with a higher level of confidence than vibes

Edit: I did want to share that if you can make something deterministic, you probably should. The slack PR example is something that id just make a simple script that runs on a cron schedule, but it was easy to pull on as an example.

In response to the idea of iterative development, it is still possible, actually! You run something more akin to integration tests and measure the output against either deterministic processes or have an LLM judge it's own output. These are called evals and in my experience are a pretty hard requirement to trusting deployed AI.

This is my 3rd google phone in a row with issues:

Nexus 6p - Had to have the battery replaced and then inexplicitly died, dead to the world Pixel 4a - Similar battery issues + a screen that physically fell out of the phone Pixel 6a - Battery woes BEFORE this update in the near future

- TLS Timeouts when pushing to ACR repositories

- Databricks queries taking exponentially longer than last week

- ADO builds hanging indefinitely until killed

- Application Insights dashboards only half loading / the other half working on a refresh

- Random, intermittent, authentication issues in the azure portal that work if tried again immediately and without change.

It feels like there are gremlins running around in Azure, but I haven't any proof outside of these experiences of mine and of other colleagues (who are working in different tennants)

Comments URL: https://news.ycombinator.com/item?id=43002742

Points: 1

# Comments: 0