What it does:

Parallel Execution: Send a single prompt to OpenAI, Anthropic, Mistral, and Gemini simultaneously. Compare the outputs, latency, and exact token usage side-by-side.

Batch Evaluations: Upload a CSV dataset to run bulk tests across multiple models at once.

Manual Diagnostics: Grade outputs manually (Pass/Fail) and assign diagnostic tags (e.g., Hallucination, Format Error) to build a human-verified performance leaderboard.

Local-first: API keys encrypted with your OS keyring; history stored in a local SQLite DB; no telemetry.

I’m looking for technical feedback. What do you think current LLM testing/evaluation tools get most wrong?

Comments URL: https://news.ycombinator.com/item?id=48570680

Points: 3

# Comments: 0

Whenever I needed an LLM to reliably output JSON or follow strict formatting rules, I kept having to write throwaway JavaScript scripts just to test the same prompt against OpenAI, Anthropic, and Google APIs to see who actually followed the instructions. It was a tedious workflow, so I built a local desktop UI to just do it for me.

What it does:

* Sends one prompt to OpenAI, Anthropic, Mistral, and Google simultaneously so you can compare the outputs in a single grid.

* Automatically checkpoints your runs. If you tweak a word and the output gets worse, you can just click back to the previous version.

* Exposes raw API responses, latency stats, and token usage instead of hiding them behind a chat UI.

* Saves your history to a local SQLite file on your disk.

* Keeps your API keys encrypted locally (zero telemetry).

Yes, it is built on Electron. I sincerely apologize to your RAM. I tried to mitigate the usual bloat by sticking to vanilla JS and native Web Components, so it idles around ~240MB, but it definitely still has some rough edges.

My plan is a one-time $29 perpetual license, but right now the public beta is completely free. I'm holding off on charging because I need a solid Merchant of Record to handle global tax compliance. Lemon Squeezy seems appropriate, but I still need to validate it (sadly, Stripe isn't onboarding new customers from India right now). If anyone has recently dealt with a payment processor that handles global compliance for desktop software, I would really appreciate your suggestions.

You can grab the Mac, Windows, or Linux binaries here: https://aptselect.com/

If this fits into your workflow, I'd love to hear your feedback, bug reports, or any features you'd actually want in a local prompt runner.

Comments URL: https://news.ycombinator.com/item?id=47618145

Points: 2

# Comments: 0

And whenever a fix or patch feel like a duct-tape to me, start looking at the architecture and keep asking myself how can this be refactor to make it more resilient.

I realized my brain naturally wants to 'fill in the gaps' with assumptions. Learning to suppress that urge and force myself to verify what is actually happening rather than what I think is happening has been the most important part of my growth.

Its a continues process, you never stop learning.

Longtime lurker here.

I built a tiny website called Latent Logbook. It's inspired by the spirit of Advent of Code, but focused on LLMs instead of algorithms.

The idea is simple: You get a puzzle (like parsing a messy log file), you solve it using any model you like, and then you share your strategy. The goal isn't just to solve them but to see how different models handle them. It's fascinating to see someone solve a task with a 7B parameter model, but it's even more insightful if someone pulls it off with a weird, tiny, quantized model running locally. (If anyone has ideas for new puzzles/traps to add, I’d love to collaborate.)

The Mechanics:

* Trust System: When you submit a solution, you get 5 Latents (points). We trust you to actually do the work.

* Archive: Solving a puzzle unlocks the shared log where you can see everyone else's prompts and models. The puzzles small and bite-sized so you don't have to burn a ton of tokens (or money) to participate.

* Upvotes: If you see a particularly clever prompt or a unique model choice, you can spend your Latents to upvote that solution.

* Technologies: Just server-side Hapi.js and Handlebars, backed by SQLite. It runs on a $5 VPS and is super lightweight.

It might crash and burn, but I think it’ll be fun to see what happens.

I’m stepping away for a bit but will be back to answer questions later.

Thanks for checking it out!

Comments URL: https://news.ycombinator.com/item?id=46135677

Points: 1

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=43740583

Points: 3

# Comments: 0