Only reason I can think why the LLM might still work better here is that it’s trained to solve a bunch of different image/video related questions, so it’s perceptual modules may be more robust adaptive for this aesthetic grading task versus something like LingBot

In the original Stable Diffusion, CLIP doubled as part of this LAION Aesthetic Filter as well as the text control for the generative model. But in the last couple of years, the field has drifted away from CLIP towards LLM hidden states (e.g. Qwen, Mistral, etc.). It turns out that text-to-image and text-to-video models do not need the visual prior from CLIP; instead the hidden states of multimodal LLMs act as better conditioning mechanisms.

But you are on-to something when it comes to leveraging pre-trained vision models as priors in generative image/video.

Nowadays, models like Dino-v3 (which have the strong spatial priors, you're talking about with LingBot) are being used in 3 distinct ways:

1) REPA: The DiT in the generative model minimizes cosine distance to another vision prior, like Dinov3. This paper from earlier this year, iREPA (https://arxiv.org/pdf/2512.10794) digs into WHAT specifically makes a good regression target for generative image.

2) VA-VAE: Most generative image/video models work in a latent space that's pre-trained separately. We wrote a lot about this in an earlier blog post (https://www.linum.ai/field-notes/vae-reconstruction-vs-gener...). But basically, VAEs that overfit to the reconstruction task create brittle latent spaces for generative models. One solution folks have figured out is to regress to a strong spatial prior like Dino, during VAE training. This is discussed at length in the VA-VAE paper(https://arxiv.org/abs/2501.01423). Recent image models like Flux 2, likely achieve their improvements through this type of VAE training.

3) RAE (Representation Autoencoder): This is the latest/greatest right now. Where generative image models are trained directly in Dino-v3 space; and then they just train a decoder to read back out into pixel space. (https://arxiv.org/pdf/2605.18324)

Hopefully this helps!

For the LAION classifier specifically, it's trained on-top of CLIP. The bottleneck for accuracy isn't the linear/non-linear readout, it's CLIP's specific knowledge of the world.

CLIP was trained with a contrastive learning objective, where all it has to learn is that images of the same category are closer to each other than images of other categories (e.g. two items are both dogs so they should be closer semantically, while say a dog and a computer are different and should be farther semantically.

As a result CLIP is really good at object definitions, but struggles at pretty much everything else. Modern multimodal LLM's perceptual modules are much stronger than CLIP; that's why we chose fine-tuning a LLM rather then training a MLP on-top of CLIP (which would be a direct iteration on the LAION aesthetic classifier).

Comments URL: https://news.ycombinator.com/item?id=49458502

Points: 18

# Comments: 8

In the past year, I’ve had 3 or 4 times when someone has “called me out” for using AI when I’m just an em dash organically.There’s no discussion of what I wrote; just immediate anger.

It feels like giving in, but I might have to edit my writing style just because of the visceral reactions

When reasoning got introduced a year ago to GPT 5, on average the model performed worse than GPT4-o for short video clip captioning (Ie hallucinating actions that didn’t happen). The old GPT 5 was extremely finicky in terms of fps sample rate.

The other SOTA LLMs (like Gemini Pro) have clearly been optimized for long video understanding, since they can’t see almost anything sub-second (even if you up the frame sampling rate).

Sol is the first model we’ve seen to accurately caption complex sub-second movements (eg woman suddenly turns heard head to right). It’s robust to different fps sample rates so I can only guess that they trained on videos sampled at different fps.

I think a simple reason why it’s been hard to unseat in Nvidia is first mover advantage. A lot more water has flown through Nvidia pipes than TPUs or AMDs chips for that matter.

TPUs and AMD chips aren’t priced cheaper than NVIDIA (at least for my purposes training models). So there hasn’t been an impetus for me to venture there and use those chips.

Anecdotally, folks I know who have tried using TPUs and AMD chips have hit more issues with the underlying drivers than with NVIDIA chips. That costs time and money to fix.

Eventually the other chips will go through enough iterations and stability will be reached

Congrats on the launch. This is cool tech

Are there any alternative ideas to JEPA right now, when it comes to speech encoding that couples meaning and sound? Curious to learn more about journey from the problem space to solution space (JEPA).

For context, in our domain video-JEPA hasn't proved to be as helpful as one would have hoped. It's decent at high level semantics (e.g. action detection) but doesn't capture enough "detail" (intentionally so) to be used as a powerful enough encoder (or regularizer). Might be just because the research models are too small / haven't been trained on sufficiently large volumes of data, yet.

So I'm just baffled, why anyone was using AI to generate comments. Like what was the incentive driving the behavior?

Comments URL: https://news.ycombinator.com/item?id=47253237

Points: 3

# Comments: 0