Currently what makes this too difficult for anybody but frontier labs is the lack of access to the full distribution of workloads/data they use to RL multiple separate envs/evals without regressing more than they advance in general capabilities. Because they are the primary buyers/builders of that stuff, they have no incentive or reason to allow anybody to replay or resample it but themselves. And it's all so very expensive to do in aggregate so there's not really demand for other product shapes except more openly available, collaborative/bundled RL that you could plug your specific workloads into (who the frontier labs would obviously not want to support with their business).

However, if someone were to build a collaborative rollout platform that you could use to train private workloads (ie create useful IP that doesn't just become profit for other labs/come from what they already can do), the upfront investment would be amortized over the very large number of potential buyers once it gets into the 5-8 figure range, who essentially have no other choice if they want to remain competitive in the technology industry.

Until openai/anthropic ipo the concentration of capital/spending and ndas/loss of employability is too concentrated for the best researchers to really do this without rocking the boat. And a lot of also-ran ai/saas have the same risk due to the lack of capitalized acquisition opportunities or AI vendors to partner with.

If the plan is ultimately to drive you out of business if you ever build anything profitable with their products, and monetize your knowledge without fairly rewarding or explaining their intent to do so, you might as well defect early and build what you inevitably would need anyway.

you could try to help build that!

Why would Walmart want to let Amazon hold their business hostage, tell them all their secrets - literally pay to hand it over to them - and generate profits that would surely be re-invested in competing against them in retail? Even if AWS handled Walmart's account in 100% good faith and helped Walmart tremendously, they would still ultimately be paying to make themselves less competitive in a world where their single competitive edge (low-margin retail) would be eroded.

Would the consumer - or really anybody that likes having cheap, widely available products they don't personally make for themselves - ultimately benefit in a world where Walmart's only choice is AWS (startup-branding) and AWS(ea-branding) and AWS(IBM branding) and AWS(Microsoft-branding, if they were also evil chinese communists who maybe trained on the traces they paid for)?

better hope you're not walmart

There’s a good chance that the next step after open weight models is something that makes it easier to post-train existing models on new workloads without running an entire pipeline just for yourself. Somebody has to actually build it though.

This is the top of the funnel for most people building a tech product or marketing their business, so it’s very helpful if you intend to sell other services they might want to use down the line. (Cf Cloudflare except imagine if they also integrated ads into one of the most popular websites in the world)

Software engineering on one end is going to be more like finance or medicine where the cost of poor performance is actually higher than the risk that they wont be worth what you pay them (ie there isn’t any amount of pay where it’s worth it to hire them). Then on the other it’ll be like knowing how to use PowerPoint. All the new tech categories have too many bottlenecks and challenges beyond just producing code to consolidate like that.

Domains, auth, and databases are probably the stickiest software products of all time. I suspect it’s a combo of security concerns and the risk of eroding goodwill among developers (their most important customer base rn) if they move up the stack too quickly. Once you know that the goal is to become cloud vendors, train on/compete with their own customers, and own the entire software stack e2e it’s hard to really feel grateful that they’re dragging it out but planning on doing it anyway.

For those of us working in infra/SaaS outside these companies it’s pretty clear that the only viable path that doesn’t involve getting cannibalized is training your own agents/models. The new coding agent-infra-data business model is a path towards full commoditization and undifferentiated prompting in 3-4 vertically integrated walled gardens. If you ever start making real money on pure software infra they’ll just be able to eat you alive by implementing something similar, training on their own tools rather than yours, and integrating it into their stack (which all of your customers are on already).

Also, you should only ever use $20-200/mo subscriptions on work you want them to train against or you don’t mind automate yourself out of. Think a little bit about what you’re teaching them to do when you use their products, especially if they’re the primary interface you’re working in. People are going to start caring about this a lot when the AI companies feel safe enough to begin the “extinguish” phase of AI coding. Train your own models!

I don't even think Cloudflare has a name for theirs, I remember hearing it called their developer platform but after checking just now, they seem to mostly just call it "products".

Devtools marketing to consumers feels pretty impossible to me. I think you either need to be absolutely massive and have existing credibility + tons of resources to throw at every possible marketing channel, or figure out how to get people to truly love your product/brand. You're basically selling trust/professional identities in most cases rather than compute or APIs.

If I were to do it again I would either pick a different vertical or put 90% of my effort into community-building, events, PR/tech-influencer stuff, and flashy demos. You only need to fall back to widely known framings like "cloud" when nobody knows about you.

Design is still hard. People are very good at sniffing out shortcuts and inconsistencies and AI “smell”. The last 10% of designing a product is more like the last 400% (because you’ll probably need to throw it away or start over). It’s not spending time on design that cheapens the product

I’m not really sure that most web frameworks reduce boilerplate. There is a lot of boilerplate involved in setting up routes, and builds, and so on. As far as I can tell they only really make sense for very large teams (who need a uniform shared data model, design language, processes, etc.) and people who really want to run server side js/do a lot of SSR.

The thing about web “frameworks” is that they’re not that different in structure from what you’d build to do SSR and a repeatable way to add pages/dirs and shared assets, in any language. You can build a minimal version of something that does that in less than an hour, at least in Go. go:embed and a factory method for handlers :)

I think we’ll all end up with more bespoke “frameworks” (specialized web servers) with less gravitational pull towards dysfunctional (IMO, the node ecosystem) tooling, and still have many people using the best major frameworks. Truly large projects will still need to use react and svelte and co unless they can train a model on their own framework. But that’s really hard

I've been avoiding investing in these kinds of tools and workflows partially because I don't understand the UX (and partially because I worry UX and tooling will change so fast that I won't get a positive ROI). Your hook sounds like a script, but your plan mode sounds like an interactive TUI, and "pull this PR and assess" sounds like a reverse proxy tool or something?

I can definitely see why the custom plan mode is useful vs just calling a script or asking the model to do something, but what's the draw to building comment cleanup and github API access into the harness vs running the comment cleanup as part of CI, or just asking the model to pull a PR and review it using gh/the web UI/the github api (with no special harness handling)? Does the model get updated with changes you make to files it already read or recently wrote?

The way I normally use coding agents is by giving them a very large task specification in a fresh session with some kind of verifiable exit criteria, and sending them off + staying out of their way. Normally I would just use other sessions or run scripts to do these things because the coding agents I've used queue any /command I give them (very frustrating when you have to wait 30+m just to check usage!) while the model is working, and in my sessions the model is working >90% of the time, so I usually have other terminal windows or applications open anyway.

Our application was behind a proxy that requires auth, and we could host our SSG on Cloudflare pages, so we didn’t have to expose any application server/compute (besides auth) to the public Internet. To me this was very desirable for security and devex: we could develop the two applications independently and be pretty lax with the static site.

All those tools and packages are constantly making breaking changes or introducing incompatibility issues or becoming obsolete, and there is so much maintenance. I went pretty deep on CSR/SSG/browser automation over the past two years and realized that everything you get out of these tools is probably 20-400 lines of javascript you could just ask an LLM to implement when you need it. I’ve seen people waste so many hours, days, and weeks on tailwind ui crap (solvable in minutes with css), vitest and object lifecycle crap (use the browser debugger, do e2e tests) and major version upgrades (there’s about 95% less of this in the browser and >99% less outside experimental APIs).

I think for Facebook/Github sized web applications, with really big teams and lots of custom tooling (eg React) these frameworks make sense. Most websites just need basic file serving/rendering, to send and respond to http requests, and some javascript to run on the client (but html and css can actually handle many core UI patterns). I’ve been much happier not using frameworks because my projects just work and the knowledge I build doesn’t constantly churn.

For most of 2024-2025 frontier models really struggled with Svelte 4 vs 5 compatibility issues. Some of the main contributors, to their credit, really put in a lot of work to create benchmarks/MCP/other tools to make AI better at using Svelte. Personally I am just not a fan of MCP and other tools like that at all, and decided I'd rather just use vanilla css/js for new projects.

Look at this; it's literally multiple times more expensive to have a model sifting through all this stuff in every context use them with a tool: https://svelte.dev/docs/ai/prompts/llms.txt All of these are input tokens and turns to gather context.

The "too niche to be a major training priority" dilemma is actually a really big problem that almost all new or emerging dev tools have now. Incumbents and category leaders are priorities and show up in major benchmarks, so models develop excellent tacit knowledge and capabilities with them and expose it everywhere all the time for "free" in their weights. Everything else is at a major disadvantage because models don't know about them, how to use them, how they work, what they're for/why, and every time you use them you pay a big capability and token hit because models only learn about them in-context.

I don't blame the Svelte team for this at all. There should be a better way for devtool projects to contribute to frontier labs training pipelines or properly posttrain their own agent coding models.

Clearly some new RLAAS/dataset/env is being used for this now (it doesn't even seem that complicated, you have one LLM judge whether gameplay is recognizable as the original game or not and another trying to implement a logically/semantically identical version of the game). It's why the performance improvement on this workload has been so dramatic.

Everything is going to go from 0->1 on this benchmark in short order because of that.

IMO it is a costly/goodhart-resistant way to “show your work” and help other people understand or challenge your mental model. (IE a justification for something you believe to be true). Overly polished writing is performative, it’s hard to take seriously once you’ve read The Economist/LW enough to see how poorly “well written” correlates with truth

To a certain extent are all wrong or ignorant about almost everything because our knowledge/time are very limited. But it is really important to understand what other people think in order to coordinate with them/align human goals and understanding.

It’s good that the average person taking the lowest-friction path to using an LLM in bad faith is easy to identify now. The more obvious and disliked it becomes, the more they’ll be hit with the stick to actually know things and not bother people. It’s so much worse to be “bad and stupid and not care” than “possibly cringe or wrong”

If they have a mostly consistent or invite-only set of attendees, the network effects don’t matter. Imagine a Boy Scout troop, Alcoholics Anonymous, pub trivia, blood drives, local alumni association, churches, user groups.

Many businesses have this problem but don’t use Facebook or discord because of the login wall and unprofessional vibes. They would pay for a real tool that makes consistent event/group hosting easy. IMO you’re getting interest/engagement in spite of the critical feedback because people who regularly hosts events need that

For example, do you think it makes more sense to aggregate groups by subject matter, or by location? Why would I want to browse events on the other side of the world/guess which category a group got bucketed into? The group/events UIs are actually good but the groups directory in the way is not, and your landing page makes the activity feed/posting something prominent for some reason (why would I care about the activity feed or want to post an update to some random site?) instead of the thing that they would actually want to use (groups/events), that actually makes an activity feed/status update make sense.

Sincerely trying to help because I've made the same kinds of mistakes before and it pains me so so much. The main reason is that I find it very hard to trust a product that seems to prioritize aggressive conversion, or strange prioritization of actual user needs vs larping as a saas, and I think it's hard to recognize that in your own work because you're thinking "I'm building a saas" and your users are seeing something that cares more about Being SaaS or prioritize random stuff more than even the basic thing it's supposed to help them do.

The impression you get isn't "more is nice to have even if it arrives in the wrong order", it's actual frustration/distrust seeing all this random stuff and what you want it do or be perceived as, prioritized over the actual thing it's supposed to do. Make it easy to find a group or trust that this is an actual thing people can use, I promise you that you don't need an Events API with embeddable widgets until you can do that.

This isn't just a complaint about the UI being "vibe-coded-coded" or incomplete, but the product itself not making the happy path very useful or confusing.

For example why is the landing page trying to sell this like a b2b saas app or hit me with a bait and switch login wall, or why is the screenshot not just the actual directory/a search bar, why do I have to know to go to groups? (Also, why are you showing me activity feeds or funneling me into that when I'm on your website primarily to look at groups and events? Why would I go to your site to look at that) Then on the group page it would be very convenient if there were just an edit/post button that I can click to draft or actually make a group (and then maybe quickly group by / filter by location or region). Same thing with adding an event, you could have a + button with a modal to draft it and only hit me with the login wall when I try to post.

The event/group UI actually looks decent but I really think you need to prioritize the core activation flow and design. Right now it makes activation, understanding what the product is, and doing the first thing I would want to do (look at groups or events) not obvious. I'm somewhat of a hypocrite on this but also speaking from experience - there's no point in spending time marketing a product with own-goals like that.

Why are your docs so much better designed/clear than the first thing I see or want to do! You don't need docs for this, it's like three database tables you can search and post to! Why the globe?? I'm not even sure why I would want or need to directly browse your users but why is there so much more effort here than in the groups UI (the thing I would be using this for)?

Since this is a pretty conventional crud app the only reason people would really have to use it is UX (or an audience you don't have yet). Trying to be helpful here because I think you should keep working on it and it's something I'd want to use if there were more attention to detail, which you could prob fix in less than a day with a coding agent.

I didn't see the article mention idempotency anywhere, and you didn't in the original response to the guy who said it just sounded like a buzzword; atomic snapshotting with deterministic execution is literally how you make a program continuation an idempotent function!

And the article is about solving the problem at the language runtime level so it doesn't even do that. So it would be reasonable to assume it is a buzzword if it does not have the essential property you mentioned and I was referring to. I was not even intending to disagree with you but just add what would make it less of a buzzword in this kind of case.