I always get a "lifestyle business" vibe when tech companies are based in LA or NY. When you're a seasoned veteran you'd prefer to live in a larger, more fun city, even though tech is concentrated in the Bay Area

If Kimi k3 really were above Fable 5 then there invariably the USG would have to consider their restrictions on model capabilities excessive, or one would have to admin closed source models are held to more restrictive safety standards than open source models.

>Although I will reiterate the fact that distillation is not the primary reason why these models are performing so competitively.

How would you know this? How could you ascertain exactly how much performance is attributable to their unique engineering/research? If they really were so competitive they could surely make a model that isn't dependent on distilling Fable or other frontier models.

>Claiming in any shape or form that fable disillation is one of the primary reasons why kimi k3 is so competitive is slandering the work of other labs that cooperatively push the open-source models forward.

If the distillation is irrelevant to why it is competitive, why do they do it then? Obviously is helps improve their benchmarks/performance to some degree, otherwise they wouldn't need to do it.

The Chinese model of model training/open sourcing only makes sense in the context of the overall strategy of undercutting American frontier labs’ profit margins.

> No, it can hold more floating point numbers.

Fallacy of composition. Just because an LLM is made up of floating point numbers doesn't mean its capabilities are limited to that of bare floating point numbers, in the same way that the individual faculties of a neuron don't preclude the human brain from emergent properties born from the synthesis of its synapses.

He claims that his company has "solved" hallucination by creating a verifiable fact-finding system, which is like saying that a person has solved plan crashes by creating a plane that never leaves the ground.

When an LLM says something incorrect, it often is due to that LLM reaching the limits of its abilities, but it doesn't "know" (for lack of a better term) what being wrong feels like, so it will try its best to fit the information it has into a compelling story. The reason why scaling leads to fewer hallucinations is that the model can hold more abstractions, more facts about the world, it can work through the complex, vague machinery of reason with more scaffolding, and more of a buffer (via its weights) to reason with nuance. This is why LLM's are useful, not because they can be fed into a fact-retrieval system, but because they can produce new information via the association of things they know.

The point is, we want LLM's to actually produce new information and work out things via their thinking, not be limited to citing facts that already exist and avoid veering into the limits of its abilities. In that sense hallucination is really just exposing the limits of scale, which would necessitate scaling models further.

Scaling is the only way we have gotten to this interesting, emergent property of LLM's. Further, the best way to make small models which don't hallucinate (that we've found so far) is to train a big model first, then distill it, or use it as a teacher to a smaller model. Either way, pursuing scale is the most defensible strategy, and a more robust solution to hallucination.

Want frontier intelligence? Better not defy the current administration, or your competitors will have access to a better model you could never use.

Like lets say i gave a human and an LLM a tech spec and the LLM created a software product as good as the human, does it matter that there was consciousness behind the human thinking?

What is your definition of thinking?