Picking specific metrics will probably not work, it would be a mix of Goodhart's law with Bitter lesson. Maybe picking and labeling quality repos, having whole suite of metrics as input features and training some traditional AI classifiers to steer the LLM training.

When I use the social platform, it is mostly as a read-only website. I am occasionally commenting, opening issues, PRs etc. The slowness and annoyances do not matter that much.

But I live in the team tool, where the annoyances and paper cuts are painful. Besides poor craftsmanship, there is often tension between the simpler design for social which sacrifices usability for pro use cases.

The social platform, which has huge network effects and is really hard to replace, is basically a loss leader for marketing the team product. I wonder if it would be viable for another company to create a team tool that stands on its own by outclassing Github like Linear outclasses Jira? That should be easier to replace. And Microsoft could keep running the social platform for the time being, where the adoption of alternatives is much slower and requires a much longer time frame.

Starting with Fable 5, if it goes off the rails, it is more difficult to correct it, because it is overall wrong, but covers its tracks with plausible sounding arguments, so it is hard to pin point and correct.

Now, this article points out techniques that were useful to rely on models more, but those peaked at 4.6. Now according to benchmarks Opus 5 is on the frontier. But when when it has looser reigns, it ends up gaslighting me even more with abstract word soup than any model before.

Related to the friction you mention, since the flow graph of the context is not preserved when it gets transformed into code, I find LLMs getting confused and making worse changes, because the explicit decisions with good reasons are getting mixed with implicit decisions from LLM surfaced during designing/planning. Sometimes when things change the code should change radically to match the new reality, but LLMs often "play it safe" by accumulating code and adding to tech debt. With a flow graph when change is made in the upper levels, it could propagate and allow decisions in lower levels to be revisited.

If you distill the feedback into a spec (probably a combination of both docs and tests) which comprehensively covers the functional requirements, then you could treat the code as cached materialization. The value of generating code again is targeting different non-functional requirements, like ability to run in different environments or catering to different performance characteristics.

The workflow I imagine is either deriving specs from the conversation or reverse engineering the code to spec, review and edit the spec which should be tighter and much more compressed, then deterministically compile to code. Of course we don't want to be spec-first only, that would be going back waterfall, but doing iterations back and forth.

Now, Claude is not a compiler because it is closed and non-deterministic (they do opaque processing on server, hiding reasoning tokens), but LLMs might be. We refer to a piece of code from npm/pip by name to get some code by downloading it. We then have lockfiles with hashes to ensure integrity. Currently we are vibing it, but in the future we could refer to a piece of code by prompt/spec and getting the code by inferring it. To ensure integrity, the lockfile would be hashes of open weights and inference code (and ironing out implementation details like non-determinism due to GPU scheduling, etc.).

What is missing in the article is the reasoning/effort levels, so it is not ruled out the results differ just due to different reasoning budgets.

I would also be interested in seeing coding performance on SWE benchmarks.

But I wonder if tmux/rmux design is suboptimal since it couples session persistence and window management together. Do you have an opinion the design which separates the responsibilities? An example that pioneered this approach is abduco, and libghostty-based zmx being a modern implementation.

With the increased volume of code with agentic coding, what was once occasional is now a daily occurrence. I would like to see new kinds of code review to deal with larger volume of code. The current Github review interface does not scale well. And I am not sure Microsoft has organizational capacity to come up creative UI/UX solutions to this problem.

One idea that comes to mind to make review easier would be to re-create commits following Kent Beck's SB Changes concept - splitting structure changes (tidying/refactoring) and behavior changes (features). The structure changes could then be quickly skimmed (especially with good coverage) and it should save focus for review of the behavior changes.

The challenge is that it is not the same as just committing the hunks in different order. But maybe a skill with basic agent loop could work with capabilities of models nowadays.

This is great work, I am looking forward how it evolves in the future. So far Claude Code seems best despite its bugs given the generous subscription, but when the market corrects and the prices will get closer to API prices, then probably the pay-per-token premium with optimized experience will be a better deal than to suffer Claude Code glitches and paper cuts.

The realization is that at the end agent framework kit that is customizable and can be recursively improved by agents is going to be better than a rigid proprietary client app.

For my agent management scripts I use zellij since it is more ergonomic than tmux. Abduco sounded good in principle, but implementation is too limited. However, zellij is quite huge in the order of tens of thousands LOC and I am using only small part of it. It looks like zmx might implement just the right amount of features for this use case, I am going to try it. It is always nice to achieve same functionality with leaner tools.

Do you also think about dvtm part alternative? I wonder if once libghostty proper gets finished it would open possibility to level up textual multiplexing and unlock some cool features with graphical UIs.

With lisp REPLs one types in the IDE/editor having full highlighting, completions and code intelligence. Then code is sent to REPL process for evaluation. For example Clojure has great REPL tooling.

A variation of REPL is the REBL (Read-Eval-Browse Loop) concept, where instead of the output being simply printed as text, it is treated as values that can be visualized and browsed using graphical viewers.

Existing editors can already cover the runbooks use case pretty well. Those can be just markdown files with key bindings to send code blocks to shell process for evaluation. It works great with instructions in markdown READMEs.

The main missing feature editor-centric command like workflow I can imagine is the history search. It could be interesting to see if it would be enough to add shell history as a completion source. Or perhaps have shell LSP server to provide history and other completions that could work across editors?