If it gets worse in future releases, we'd likely step fully away towards more useful (for us) models even if they're less capable.

I think that "deciding what types of code can be reliably handed off to AI" might be missing from the list. It's orders of magnitude easier to nail 80% all the time than 100% all the time. I could see standalone products even developing in this space.

In practice it's pretty difficult to find the balance between yak shaving and piling in unnecessary manual labour by just trying to do the work with existing (possibly poorly fitting) tools.

If you're planning to stick with your current tools for a long time, each 1% improvement compounds massively over time, so that balance is probably much closer to yak shaving than most people might realise.

My (boring b2b/b2e) org has scripts that wrap a small handful of agent calls to handle/automate our workflow. These have been incredibly valuable.

We still 'yolo' into PRs, use agents to improve code quality, do initial checks via gating. We're trying to get docs working through the same approach. We see huge value in automating and lightweight orchestration of agents, but other parts of the whole system are the bottleneck, so theres no real point in running more than a couple of agents concurrently - claude could already build a low quality version our entire backlog in a week.

Is anyone exploring the (imo more practically useful today) space of using agents to put together better changes vs "more commits"?

Do you have customers who have faced/solved this problem? If so, how did they do it -- it seems like a killer on the approach?

I'd love to see it feel a bit more polished on desktop (maybe I'll give that a shot if I find a bit of spare time!) - I could see a few simple things like adding up/down arrows to the picked item and wiring into up and down arrow presses going a long way to making it work really well there too.

Genuinely, thank you for sharing this, it's something different and interesting.

RAG could largely be replaced with tool use to a search engine. You could keep some of the approach around indexing/embeddings/semantic search, but it just becomes another tool call to a separate system.

How would you feel about becoming an expert in something that is so in flux and might disappear? That might help give you your answer.

That said, there's a lot of comparatively low hanging fruit in LLM adjacent areas atm.

While it evaluated the xslt serverside it was a really neat and simple approach.

Until the tech catches up it will have a stifling effect on progress toward and adoption of new things (which imo is pretty common of new/immature tech, eg how culture has more generally kind of stagnated since the early 2000s)

In a research context, it provides pointers, and keywords for further investigation. In a report-writing context it provides textual content.

Neither of these or the thousand other uses are worthless. Its when you expect working and complete work product that it's (subjectively, maybe) worthless but frankly aiming for that with current gen technology is a fool's errand.

If they choose to follow orders they know are illegal they can be personally liable.

I can write something with Microsoft tech and expect it with reasonable likelihood to work in 10 years (even their service-based stuff), but can't say the same about anything from Google.

That alone stops me/my org buying stuff from Google.

> we use a stricter evaluation setting: a model is only considered to solve a question if it gets the answer right in four out of four attempts ("4/4 reliability"), not just one

This surely makes the other models post smaller numbers. I'd be curious how it stacks up if doing eg 1/1 attempt or 1/4 attempts.

- write some moderately complex powershell to perform a one-off process

- add typescript annotations to a random file in my org's codebase

- land a minor feature quickly in another codebase

- suggest libraries and write sample(ish) code to see what their rough use would look like to help choose between them for a future feature design

- provide text to fill out an extensive sales RFT spreadsheet based on notes and some RAG

- generat some very domain-specific realistic sounding test data (just naming)

- scaffold out some PowerPoint slides for a training session

There are likely others (LLMs have helped with research and in my personal life too)

All of these are things that I could do (and probably do better) but I have a young baby at the moment and the situation means that my focus windows are small and I'm time poor. With this workflow I'm achieving more than I was when I had fully uninterrupted time.

Each of the ideas in here is solid, but there's too much writing around the core idea -- a sentence or two for each point and then a tldr like "put in some basic level of effort if you're going to ask for others' valuable time." would do it for me personally.

Anecdotally, most parents within my circle bought their Snoo used and sold it after use. I bought an unopened snoo from facebook marketplace for $X and sold it after 6 months for $X-200.

I was a little annoyed that Happiest Baby is meddling with the resale value (because I was expecting to be able to sell it on after a few months of use)

IMO even though the product is overpriced, I'd have happily paid 5k for the extra sleep I believe it gave me.

It works for us probably because we sidestep the pain points you list - the environments we run in are pristine complete copies of known datasets, we remove as many sources of randomness as possible, and our environment flakiness level is very low.

They still break but usually because the locators in use have been chosen poorly (or we've made planned changes to a page/component)

We're a web based b2b saas that runs an instance of the entire environment for each of our customers. Our non prod setup consists of a bajillion static test environments but more importantly we use testcontainers to spin up the transient test environments from database snapshots. Using the recorder on the static environments (before the transient ones existed) _was_ a pain

"It's dangerous" is a beneficial lie for eg openai to push because they can afford any compliance/certification process that's introduced (hell, they'd probably be heavily involved in designing the process)