I'm working in this area recently and very keen to use this given the claimed performance benefits, but I tried all your links and didn't see any actual performance numbers. Do you have any to share?

IMO a fair performance benchmark for those not tied to the full pytorch stack would have ffmpeg and the wav already loaded into memory before execution. Given that torchcodec relies on the user-supplied ffmpeg installation I suspect that may not be the case for ffmpeg already, at least not by default.

I understand why meta wouldn't want to do this (then you are inevitably distributing exploitable security vulnerabilities in pytorch, because ffmpeg will probably always have them) but I've been statically linking fmpeg and keeping the binary in-memory while still using separate processes for different batches of audio, with I/O through UDS between the parent and ffmpeg; then the parent does VAD on the pcm on CPU before any further inference. My implementation for static linking is similar to the pattern in https://github.com/amenzhinsky/go-memexec#static-binary - would be interesting to see if this is possible in the pytorch/python ecosystem, or maybe it's already been done.

RSI within a top-down, concentrated power structure creates an unstable equilibrium and will instantly devolve into an epic power struggle. If you control the company/computers/military with the one most powerful thing, you control everything else. It will probably just be seized but then you instantly become a target to all the other dispossessed/scared/opportunistic people who think they would do a better job than you, or just want it. It’s a stupid power fantasy to think that people with guns and will just let you reign from above as some kind of benevolent researcher-king with absolute, unaccountable control over the economy. Even worse if there’s mass joblessness and nothing else keeping most people busy.

If we can build a horizontal, federated (not the performative kind like Bluesky, has to actually be performance competitive and distributed without jank) intelligence on a mix of commodity and specialized hardware - which BTW is exactly what even Anthropic would have to work with too, a bunch of datacenter gpus of different generations + traditional compute + edge compute/network devices, maybe some ASICs, then finally consumer devices and webgpu - then there is much less risk that AI will be used to concentrate power or amplify bad actors (without their actions being immediately reacted against by the overwhelmingly larger set of good or neutral-with-something-to-lose actors).

The main barrier to federated learning is figuring out how to economically structure this, it has to have a self-funding mechanism that is hopefully more grounded in actual value than something like crypto where it’s purely forward-looking demand (/speculation) or artificial/enforced scarcity. Also it obviously has to be secure, but the risks are different vs companies like Anthropic that are trying to guard their IP - in this case it’s mostly just protection from bad actors trying to pollute training data in a way that would only be noticed after it’s expensive to fork away from, plus just generally using it as a malware distribution or data collection mechanism.

I guess I’m looking for a kind of bulk/sticky dropout (which was in fashion way back when I studied DNN in school).

It's much harder to RL out design taste because it's not self-grounding, and human labelers have no real skin in the game, so this (having a human with a vested outcome in the process directing a model's work) is the best way to get LLMs better at design/"taste"/aesthetic judgment themselves. We were working on the same thing 7 months ago and then I realized that winning over designers to do this would be a huge uphill battle setting up an inevitable fall from grace later on.

What makes me most suspicious of Claude Design is that when you disconnect and reconnect later, it loses context and nags you that the product doesn't work like that. Bullshit. It's at best an anti-abuse/implementation detail (to keep you from launching 10 at once and coming back to them later) or product shortcoming that just so happens to be optimized for keeping you from continuing your design in better tools than theirs for the inevitable followups.

It's great for one shots and it makes sense when you're trying to build a vertical product development stack like Anthropic but I'm disappointed it feels more like a tool optimized for keeping you in their product than for what you're working on. If a company other than Anthropic had shipped this - it's not that hard to build a visual self-eval loop, just use Chrome Devtools Protocol to run headless chrome and take screenshots -> feed into a judge LLM for feedback -> continue - I don't think it would really have seen much adoption.

That said, AI trained on Actor-Critic with a tight human feedback loop definitely seems like the right approach to solving the problem, just not something I want to spend my time training for someone else unless I can do so with higher "entropy" ie high parallelism/optionality

Most other founders/business owners and investors I know don’t see that as a controversial statement (that it makes no sense to see access to capital through a scarcity mindset, when it is factually accessible) but because most customers or potential employers aren’t one of those, it’s been a problem because this isn’t what you’re “supposed to do” and so they read into it from a social/legitimacy angle.

Regardless, it’s quite rewarding IMO and I highly suggest it to people with the means to pursue that path. I don’t see why people get so worked up on the whole VC thing, at the end of the day it’s just lending. Having dabbled in angel investing on the end you get a lot of people lining up for what they clearly perceive as unsecured loans or a social signal and on the other so many people get caught up in the dynamic of dangling said unsecured loans (when I first started it felt like some of the investors reaching out to me were just doing it to boss me around or something, like sir you can clearly see I started this two months ago and you dm’d me on LinkedIn to chat, I don’t need your money).

IMO most good founders/investors are credibility-maxxing but because of the social dynamics and moral hazards inherent with spending OPM you get weird other behavior

I don't think it's as simple as saying China's hosting is subsidized, they have generally cheaper electricity and labor costs than in the US and don't have access to the top tier models, and a large internal market where the big models are the best thing they can run with what they have. So obviously they max out on their top models (which are trained with their hardware market in mind, not ours) and get the economy of scale from that, and can run generally the same hardware for less money than in the US because

The edge models are very cheap to run and can do so on inexpensive hardware. They are like 95% cheaper to run than Haiku, so the math is in their favor for certain batch workloads. Most people just run the models for themselves when they do that without making it available on openrouter or whatever, because you can just provision a gpu node and use it as needed, and it's not that expensive to run this family of models.

Is your problem that you want to call Chinese models hosted in the US because you're worried about the data handling?

At my prior job there was still what felt like a strong enough correlation between my actual performance and my pay that I don't think I would have had a hard time justifying the expense there either; now I absolutely don't. With the current state of the models, it's baffling to me to hear about professional software developers planning their work around their $20/mo subscription's quotas.

Obviously it's more complicated than more tokens = more productive, but I see them less like SaaS and more like gasoline, where if I run out or need more to do what I'm doing, as long as I'm not being wasteful, I just buy more. Why would I waste a day walking 30 miles by foot when I can just pay $5 for gasoline and drive?

The most I’ve ever spent in a month extra on API tokens for my own work is $200, and I pay for the $200/mo Claude. I use these models quite a lot, though not idly (I usually just walk around and do other stuff until I know how im going to approach the next set of problems). So it costs me about $3000/year to get as much as I want of the best model available. Already that seems low enough to not be worth stressing out too much about optimizing it, because it feels like an indisputable good value, and trying to save money with a less powerful model would be optimizing for a $1000-$2000 saving at the expense of a large portion of my work taking longer or being more frustrating and iterative.

That’s not a flex or anything, I get that in other countries $3000/yr is a lot of money for a software developer and also a lot of people would perhaps rationally be better off doing X% worse at work or spending Y% more time on tasks to save $Z, if their productivity improvements didn’t translate to more salary. Otherwise if your performance has more upside I really do think that the smartest models are better with the current pricing scheme. Deepseek and the other Chinese models spend a LOT of time thinking, and tend to be much more jagged (benchmaxxed) in performance. How can dealing with that over an entire year be worth $2k?

The only situation I can think of where sacrificing my own time/performance to save on inference is batch compute (of course, $1k vs $100k is different from $30 vs $3k) or work where the tier 2 models have crossed the “good enough” threshold. But I think Opus is not even close to that threshold generally yet. As it gets smarter I, and I think most others probably, just try to do harder things faster and hit the next wall.

"If I am not for myself, who will be for me? If I am only for myself, what am I? And if not now, when?"

This inspired me to seek out more about Rabbinic Judaism and its theology more deeply, and I found the language and analogies concerning the idea of "repairing the world" (which you referenced, but which I think at first glance aren't necessarily something most people would identify as a specific core doctrinal theme) particularly inspiring [2]. To me it's frankly beautiful and something I recommend anybody interested in metaphysics or ethics/morality looking into; it also ties into the Kabbalah. IMO this aspect of Jewish theology deserves to be more widely known because it's something all of us can learn from.

[0] https://www.gnu.org/gnu/thegnuproject.html

[1] https://en.wikipedia.org/wiki/Hillel_the_Elder

[2] https://en.wikipedia.org/wiki/Tikkun_olam

It’s ok if you don’t want to do the same kind of thing but I find it weird how dismissive so many people get about wanting to use LLMs for large projects, or how anybody who says they’re using them for these kinds of things (I’m doing similar for other stuff) gets challenged on what they’re doing it for.

My use case would primarily be in search, integration, and indexing other software projects with my own, as well as transcription/indexing of interesting video and audio content (eg Dwarkesh interviews) that I don’t have time to watch but want to easily search and apply to my projects, and search/indexing for useful information from things like Linux kernel and security mailing lists. Basically there is a lot of stuff that, if the cost were low enough, I would point a reasonably intelligent AI at to distill out useful information and apply it to my projects, or just cherry pick the interesting things out and surface them to me so I don’t have to wade through all the mundane stuff and man-made slop getting in the way.

Once enough gaming compute runs at the edge it also allows for more technically advanced games than would currently be economically feasible (but aren’t made mostly for lack of a market/adoption of cloud gaming and the resulting lack of technical know-how). So I think it will stick and probably end up winning over the holdouts, once the cost of rendering the games they want to play with consumer hardware becomes too large to stomach.

In a world where TSMC is functionally capable of the same level of production but not in such a complicated geopolitical situation regarding semiconductor manufacturing, things would be quite different.

There’s a lot to criticize Sam Altman for saying or popularizing culturally but I’ve come to think his “this is the worst it will ever be” is, in the long run, actually a very intriguing and underrated point.

In a decade training LLMs to the current level of sophistication, which is in my opinion rather advanced and probably has lots of additional upside just from constructing better RL training regime independently of hardware advancement, will become just as table stakes as running a database is now. I highly recommend everyone look into the Allen Institute’s projects in GitHub and HF because they have open source training materials (including an LLM from scratch off common crawl, and some quite interesting tunes of qwen) to get a taste for what will be in the near future afternoon projects or educational material. The future is going to be wild

Unfortunately since it's not FOSS and there's no information about the company/individuals behind it, or even a way to pay for it/get licensing information from your UI, there is absolutely no way I would download it as a binary and run it on my computer as you suggest. That is, IMO, incredibly sketchy