The point being that sometimes the tools themselves don't need to survive because you take the lessons from one thing to another (e.g. move semantics and rust/modern c++)

[1] - https://pragprog.com/titles/btlang/seven-languages-in-seven-...

Do you have further insights on AI and education since?

This applies to all software projects except for the "useful to you" part, which turns into valuable to "your/the business"

[1] https://youtu.be/c8AzqMr87gQ

It had such a pleasant interface for that time.

But yeah, one of the things made me go for vim over Emacs a long time ago was its relationship with touch typing and not leaving the home row with the vim modes.

Learn Vim script the hard way [1], even if you didn't end up writing any actual vim script, was a game changer in terms of understanding the semantics.

[1] - https://learnvimscriptthehardway.stevelosh.com/ was

I looked superficially at your site/repo and based on that initial impression:

- Your approach of comparing different parts of the "black box" which affects agent behaviour (Harness, foundation model, skills, context (in your case the loaded on AGENTS.md context) is closely aligned with how I both think and operate. - You're both tackling the "regression" and the "answer hypothesis easily" problems.

> Stalking your profile (sorry..) I see you're pretty deep in the eval space, so I'm super curious what your approach has been to being rigorous for things like skill changes?

It depends on the level of automation and risk profile. For skills I use this framework of thinking [1] and encourage evals/ground truth as soon as possible so that you can have automatic feedback loops for the markdown part and for the deterministic part (scripts). Once you have the eval/ground truth pair, you're almost doing TDD or Eval Driven Development (which is quite hard the first times you try and realize you actually need to think about intent). The scripts should definitely have their own unit tests for the "skill iteration" in the event that a mutation is desired to cover new behaviour/fix wrong behaviour.

On Agent Skills, it may seem tempting to want more "openness" for the AI to solve the problem creatively but, more often than not, you've described a repeatable workflow and you want predictability and stability instead of novelty so it's really about 1) How can I freeze it to keep being good enough as much as possible 2) How can I know if something happened somewhere which changed the black box (e.g. coding harness auto model picking screws things up 3) How can I make the skill itself ETC (Easy to change), to keep control. Local Models can be a great tool for stability in some scenarios.

In particular, I prefer pass/fail (binary) outcomes instead of scoring which doesn't help regression decisions. Defining "good enough" should be very clear. Flakiness is not a good thing to accept, if the outcomes are consequential.

Anything actually risky should be solid RBAC/policy which doesn't really depend on the LLM.

I had a site that I didn't manage to make visible in HN to create a community for ai-evals.io. I've since interacted with a few people, developed further insights and given some private talks but need to get back to publishing outfacing and trying to contact more people interested in this space because it's absolutely critical. There's a lot of nuance in how different environments think about the eval problem differently: It's all about tracing and course correcting after launch, it's about simulations, sandboxing, security, automatic eval generation, etc.

In any case, I'll try to be more present from now on, and especially from June onwards to try to exchange insights in the open with people who are exploring different solutions in this space.

[1] - https://alexhans.github.io/posts/series/evals/building-agent...

That's fair but I think this is similar to power tools like vim, obsidian or others. There's the path of grabbing other people's workflows and not being able to modify them to really tailor the tool to your needs and there's the minimal incremental path that empowers you and gives you control all the way through. It gets you to understand the tools and you'll be able to think possibilities that match your exact problems.

I'm not dogmatic about it but I do really recommend it. You can see the transformative shift once people start "skill building" instead of "skill consuming".

Edit: The approach I mention works with non engineers/developers. So there's no different technical bar.

Conceptually, you should treat them as incremental software instead of magic you grab from others [1]

The killer feature is that coding harnesses tend to have SkillBuilder agent skills so creating them becomes very easy and you can evolve them.

I recommend you build your own for your particular pain points.

Very simple example [2] showing what another user mentioned around "evals" so that you can really achieve good enough correctness for your automation.

- [1] https://alexhans.github.io/posts/series/evals/building-agent...

- [2] https://alexhans.github.io/posts/series/evals/sketch-to-text...

Where are you keeping your design? If it's a WYSIWYG doc like word, what would it take for you to switch to a more text driven world?

If you don't force yourself to see it as a scarce resource it's very easy to end up in situations where it takes up most of your time.

Comments URL: https://news.ycombinator.com/item?id=47931358

Points: 2

# Comments: 1

You're waiting a whole week for an update or to push people.

I'm a huge proponent of "async over sync" and "sync as soon as necessary". Don't wait to the next meeting in 3 days to escalate.

People whose calendar looks like meetings only and are always willing and able to throw complexity to others won't feel the pain of not having a meeting budget [1] but other roles will want to avoid eating up all their time on this type of admin.

I encourage people to explore how extremely simple Agent Skills workflows can already do a lot of the admin legwork than an executive assistant would and free their time from admin bureocracy and red tape, as much as possible. If there's no programmatic endpoint or MCP, use something like playwright. A little goes a long way.

- [1] https://alexhans.github.io/posts/meeting-budget.html

If you can define good enough for you and local llms can meet that you'll get:

- no vendor lock-in (control)

- price

- stability (you decide when to hot swap with newer models)

- speed (control)

- full observability and predictability.

- Privacy / Data Locality (Depending on implementation of infrastructure).

- [1] https://alexhans.github.io/posts/series/evals/measure-first-...

Both are useful to different people (and role families) in different ways and if you don't feel certain pain points, you may not care about some of the value they provide.

Agent skills are useful because they're standardized prompt sharing but more than that, because they have progressive disclosure so you don't bloat your context with an inefficietly designed MCP and their UX is very well aligned such that "/SkillBuilder" skills are provided from the start and provide a good path for developers or non traditional builders to turn conversations into semi or full automation. I use this mental model to focus on the iteration pattern and incremental building [1].

[1] https://alexhans.github.io/posts/series/evals/building-agent...

https://alexhans.github.io/

No need to read them, just a vibe check would be insightful. It's weird how branding, even before AI had a lot of the same catchy patterns, and now it's hard to define what is the right prose (engineers might want one thing and other role families others) and sometimes you're trying to write almost with the "everyone else bag in mind" because that's the personal connections you link your "here's what I often repeat in written form".

For those that don't know Armin, he's the creator of flask among many things and has some cool projects like around like gondolin.

As for pi, you should absolutely try it out if you've used opencode/continue-dev or even if you've used the less open cloud vendor coding harnesses like Claude/Codex.

I do agree that they write in a way that requires context.

- [1] https://open.spotify.com/album/1R6rh9My8CTK4DqZorJR0V?si=3Ct...

If you have specific song/interpretation recommendations I'd love to hear them.