I understand the current hardware limitations and that you can't just put a frontier LLM in a black box and hook it up to your existing MBP via USB-C. In my estimation, something like a Apple Mac Studio M3 (256gb or more of unified memory) is maybe one possible option ($7,500 - $10,000) for running a 405b open weights model... but it wouldn't be very fast. And it wouldn't come close to the level of quality or workflow of Claude Code.

To really run a current frontier LLM locally with something like >30 tokens per second would probably require four A100s.. add in NVLink bridges, expensive cooling, 256GB RAM, a cool case with LED lights (optional) and we're talking about ~$60,000? $80,000?

So my question is: How many generations—or what specific architectural shifts (specialized ASICs, better quantization, etc.)—do we need before we can buy a dedicated co-processor box that sits on a desk and runs a Sonnet-level agent at viable speeds... at a price point where it makes sense vs. spending $500-$2,000 per month per developer on API fees? In my opinion that "makes sense to me, here's the credit card" price point might be $10,000 right now, but I could be wrong.

And related question: Who will do this? Anthropic could probably make a killing right now IF they had could sell "Claude Code in a box for $10,000" but would they ever want to? It would be cannibalizing the majority of their business. But Apple might do this. And it might only be one or two generations of hardware upgrades away. They just need the "frontier LLM" to stick into the box.

Comments URL: https://news.ycombinator.com/item?id=46356220

Points: 2

# Comments: 3

It's basically what I've come to expect from most Google products at this point: half-baked, buggy, confusing, not intuitive.

The AI companies are signing deals with large media and publishing companies to get access to data without the threat of legal action. But nobody is going to voluntarily make deals with millions of personal blogs, vintage car forums, local book clubs, etc. and setup a micro payment system.

Any attempt to force some kind of micro payment or "prove you are not a robot" system will add a lot of friction for actual users and will be easily circumvented. If you are LinkedIn and you can devote a large portion of your R&D budget on this, you can maybe get it to work. But if you're running a blog on stamp collecting, you probably will not.

The issue is not the technology. When it comes to natural language (LLM responses that are sentences, prose, etc.) there is no actual standard by which you can even judge the output. There is no gold standard for natural language. Otherwise language would be boring. There is also no simple method for determining truth... philosophers have been discussing this for thousands of years and after all that effort we now know that... ¯\_(ツ)_/¯... and also, Earth is Flat and Birds Are Not Real.

Take, for example, the first sentence of my comment: "Whenever I see one of these posts, I click just to see if the proposed solution to testing the output of an LLM is to use the output of an LLM... and in almost all cases it is." This is absolutely true, in my own head, as my selective memory is choosing to remember that one time I clicked on a similar post on HN. But beyond the simple question of if it is true or not, even an army of human fact checkers and literature majors could probably not come up with a definitive and logical analysis regarding the quality and veracity of my prose. Is it even a grammatically correct sentence structure... with the run-on ellipsis and what not... ??? Is it meant to be funny? Or snarky? Who knows ¯\_(ツ)_/¯ WFT is that random pile of punctuation marks in the middle of that sentence... does the LLM even have a token for that?

edit: typo

Comments URL: https://news.ycombinator.com/item?id=39470537

Points: 1

# Comments: 1

1. If you have capital to invest, you could do worse than AI startups at the moment.

2. Nvidia's long-term threat is not just direct competitors (AMD, Intel), but the big cloud-players going to in-house chips. Supporting the next wave of your customers makes sense.

3. Using Nvidia is the path of least resistance right now. If you only invest in startups using your products (and you are an active investor), you give startups another reason to avoid taking a risk on the alternative.

edit: typo

On the one hand... Companies spent the past couple of decades engaging in various SEO hacks to rank high on search results and OpenAI scraped the internet to train a language model. Theoretically, it seems possible that some of the SEO techniques at least partially colored the flavor of LLM-generated text, and an "AI detector" could pick that up. So if you do a great job writing SEO optimized text (wordy, structured, lots of repeated key words, etc.) you are more likely to be flagged.

But really.. "AI Detector" services are snake oil and will lead to the creation of "Anti AI Detector" services that offer protective spells against the original snake oil. See, we eliminate a bunch of jobs with AI but we create whole new disciplines of work that didn't exist before. "AI Generated Content Obfuscation Specialist - III - W2" coming to a job board near you soon.

https://youtu.be/Q7ERXm9OwuE?t=250

Also, the music doesn't sound "Lofi" because it's generated by algorithms. A lot of hard work and software goes into taking a clean, pitch-perfect digital signal and making it sound like something playing on a record player from the 70s.

Question: if the goal is to build a knowledge graph that can "explain how anything in the world is related to everything else" how do you measure progress toward that goal? And how do you measure the quality? Just having a bunch of topics and relationships is not a great metric in my opinion. Obviously this is still very early, but here's an example I found in about 30 seconds of clicking around:

"Evidence suggests that Heart Failure is related to Income and COVID-19." [https://www.system.com/view/topic/P0XELnR0PaK]

There are topics in System for "Obesity" and "Smoking", but those are not associated to Heart Failure.