AI small enough to run locally on a 30 dollar smartphone is far enough from useful to be effectively null.

Not only that, but AI is categorically terrible at ux review. I don’t mean aesthetics, I mean attention flow, workflow in general and what the user anticipates as a logical path to feature discovery and exploration. Increasingly, features are hidden behind workflows that meet a single use case but fail to be universally logical or in many cases, even accessible at all outside of the test case.

So, people… use AI in code review. It’s actually pretty great at it if you do it right, and you can ship better software with fewer bugs than ever before. But please, don’t pressure your ux guy to “10x” with AI. In fact, now that you are “10x”ing your development cycle, you are probably going to need to hire -more- ux developers if you don’t want your product to feel like slop even if it’s gold under the hood.

Just putting 100 of the smartest people in the planet in the same room doesn’t mean profit. It’s the underwear gnome problem. AI is no different in that regard.

I expect I’ll be able to run 1/2 terabyte models in 5 years for a couple $K in hardware. Inference is going to be a lot like Bitcoin mining. Only the peak of the power/performance curve is going to be profitable. Which means a glut of surplus hardware that can’t make money but makes tons of tokens.

What I meant though wasn’t agents building UX. I meant that in our case at least, the agent is the UX. “put a new extension 117 for Sam at the port next to Jane’s, and have it ring through to his cell when he’s not in the office. Leave the old 117 as is but only ring where Sam is, if that is known” basically, agents just do everything with SSH, and you can talk to them instead of poking around with a mouse. We don’t build ux for internal systems anymore. Local inference is strong enough to be reliable now with the correct tools and policies, and for 10-20k in hardware you are self contained. (Obviously $more if you are going to have a lot of people using it at the same time)

The “one wierd trick” is to build testing jugs for everything. Before a configuration is deployed, it’s tested against a toy instance, all existing functions are tested in all common cases, new functions verified, inputs fuzzed, adversarily abused, and stress tested against dummy loads. If its code, it gets examined against development policy, self containment, explicit interface walls, race conditions, code standards, complexity, factorability, and often A/B testing of different architectural choices. Except cryptography and a couple of other things, we generally prohibit the use of any libraries, it’s just not needed and the code never passes our tests anyway.

People are just YOLOing it and assuming that if it basically functions and looks right then it must be perfect. You’d have the same luck if you told your UX guy “test only the obvious, and if it looks about right at first glance, merge it. Don’t do extensive testing.”

The real thing with AI is that now you can afford to do a ridiculous amount of automated testing, setting up test fixtures, hardware simulators, mannequins of all the external interfaces, fuzzing, etc. Our testing code is 2-3x our product code, and we are shipping 10% of the bugs we shipped with humans, and it’s still 1/4 the cost and 4-10x the speed.

People shipping all the crap that is everywhere now have bad process engineering. Plain and simple. If you try to build software with AI like you used to with people, you’re going to have a bad time. The pipeline is very different.

The whole process hinges on one thing. You have to specify your code. Specification is code at its highest level of abstraction. So you are writing software by specification. The LLM pipeline is compiling it into code.

LLMs are useful in a back and forth to create the specification, but that’s where the human work is.

You don’t write software with a prompt. You specify an application, a user story, the protocols, and the API, then the LLM turns that into good code in a highly iterative and automated process. There is some human input needed during compilation, but not much.

The review pipeline is automated. Human review is rare.

The whole process hinges on one thing. You have to specify your code. Specification is code at its highest level of abstraction. So you are writing software by specification. The LLM pipeline is compiling it into code.

LLMs are useful in a back and forth to create the specification, but that’s where the human work is.

You don’t write software with a prompt. You specify an application, a user story, the protocols, and the API, then the LLM turns that into good code in a highly iterative and automated process. There is some human input needed during compilation, but not much.

Right now I’m running 2x 3090 and 128 gb ram on 3.8 flash-next180b (the quant is 97gb) and it hums at120t/s. It’s really a surprisingly good model for general tasks, and I’m soon to see how it does with more agentic kit. For agentic work it does slow down quite a bit when the context fills up… but what do you want for running a 180b model on consumer hardware?

Right now I have it set up as an administrator for our asterisk PBX and radius server, and it does higher level judgment on our cctv footage for notifications to security. It’s been really good so far. Agents are the new UI, everything just works over SSH now.

The difference is testing. Testing at every level. If crap code can’t move down the pipeline, you don’t get crap code. If a company is used to YOLOing code out the door with human developers and they try the same thing with LLMs they are going to have a bad day.

It’s basically an office of persistent agents with different defined roles and goals. The gatekeepers whole job is to refuse code. The engineers whole job is to write code that will pass. Each significant component gets its own engineer agent. There is an office manager that coordinates the agents, maintains the project documentation, keeps up the job boards, maintains scripts to keep agents pointed at the relevant onboarding documents As they come back from compaction, and manages office automation in general. Ther is an agent in charge of making sure the test coverage is as close to fully tested as possible, including refusing code that is not properly testable.

If it’s a big project, there may be an information management agent that manages the technical side of the documentation and maintains a searchable library, like an oracle. I may put in a personal assistant to help me to prioritize my involvement. Many projects also include an agent that creates mockups, jigs, and simulators so that tests can be run on code that needs a context.

I maintain another office just for research, because research can seriously sidetrack engineering. It has all of the standard staffing, but its whole job is to do research and write reports for other teams.

Agent persistence is maintained by having agents write down memories learnings and musings(not work related) and curiosities for the next instance to read in the onboarding for their seat.

Ethos, standards, practices, and policies are also in the onboarding process.