Essentially all I needed was a way to upload a data set, run tests against that data set and spit out a percentage of pass fail.

Braintrust makes this pretty easy, but If I was to do it again I would vibecode the same functionality.

It wouldn’t be too difficult to build something like that for your own usage, but I found it pretty easy to get datasets set up.

Essentially a game changer in understanding if your prompts are working. Especially if you’re doing something which requires high levels of consistency.

In our case we would use LLM for classification which fits in perfectly with evals.

Any competent designer gets really good at justifying their decisions. Everyone has an opinion about design and thinks that their taste is correct.

I’m glad I don’t have to deal with that on the software side.

Also, I don’t think I have ever applied to a fake job.

I’ve always prided myself in being an honest but considerate person.

A recent experience with a colleague who weaponised my honesty in an attempt to manipulate me has left a foul taste in my mouth. Luckily their contract ended and the problem resolved itself.

But I remember distinctly feeling that I will be professional and polite but I do not automatically owe anyone my honesty.

There are an infinite amount of problems to solve.

Deciding whether they’re worth solving is the hard part.

I’m not sure if that would work in your use case, but you could classify each line into a value using an LLM then hard code the trends you are looking for.

For example if you’re analysing something like support tickets. Use an LLM to classify the sentiment, and you can plot the sentiment on a graph and see if it’s trending up or down.

A lot of software has friction to get to the value. This is often because of constraints not choice.

To give a concrete example of this, in my company we had users upload files for analysis. To get the export for the file, it took many steps. Not hard, but a lot to get done.

We switched it to an integration and now it’s 3 clicks. We’ve gone from 10% of users onboarding to 100%.

It doesn’t mean we get people to stay, but the barrier to understanding if our tool provides value to them has completely disappeared.

I’m very curious though, what value did you strip away when trying to make your product easier to use?

I see this a lot with developers who come in and start to criticise before understanding.

There is always a reason for why something is as it is, and it’s unlikely that the people before you were just idiots.

You get buy in, start having conversations see what AI people have explored. Have they tried claude? Do they prefer other tools? If so why? What are the objections. Actually listen. I’d also showcase what you can do. I love to present what codex has found when debugging something, or a prototype I’ve put together.

If you have the budget pay for subscriptions so they can play around.

Also, you say that development velocity is a big problem, but I would dive into why that is. You may be disappointed when velocity remains the same with AI tools.

https://acronyms-suck.com

The opposite applies too, if you try hard enough you’ll find plenty of similarities with an Eskimo. We’re all just people in the end.

Sometimes it’s super obvious because a game company steals code from a previous employer, but I have never seen this play out in entreprise software.

Accounting is quite broad and there are many examples of where LLMs can help.

For example, what is tax deductible is subjective to local laws. This is essentially a classification problem that LLMs are particularly good at.

For example, most of what you describe can be solved using Xero in Australia.