You could spend 80% doing the work and 20% writing about it, or 99% doing the work and 1% copy-pasting Claude's writeup about it into a blog.

There is nothing wrong with writing if you are into it, and yes you can probably do better than Claude, but I can related to engineers who just want to build.

In this case, google has already done it, and that will be true for high resourced accelerator companies like Google working with the most popular operations like attention.

As long as you use those operations, you are okay. But if you do something different, you need to be prepared to do all of this yourself.

I think companies that don’t navigate these correctly eventually lose.

You may not agree with it, but I appreciate that it exists.

I don’t think it will last among researchers who think beyond production LLMs

It did make money that would be interesting to a startup, but not to a public company.

Some of the near misses I remember included bitcoin. Many of the other attempts didn't ever see the light of day.

Luck in english often means success by chance rather than one's own efforts or abilities. I don't think that characterizes CUDA. I think it was eventual success in the face of extreme difficulty, many failures, and sacrifices. In hindsight, I'm still surprised that Jensen kept funding it as long as he did. I've never met a leader since who I think would have done that.

Let's say X=10% of the GPU area (~75mm^2) is dedicated to FP32 SIMD units. Assume FP64 units are ~2-4x bigger. That would be 150-300mm^2, a huge amount of area that would increase the price per GPU. You may not agree with these assumptions. Feel free to change them. It is an overhead that is replicated per core. Why would gamers want to pay for any features they don't use?

Not to say there isn't market segmentation going on, but FP64 cost is higher for massively parallel processors than it was in the days of high frequency single core CPUs.

It makes sense if you are collaborating over IRC, but I feel the need to face palm when people sitting next to each other do it.

What is your preferred way to talk to your team?

No English, only code

Slack

Zoom

In a meeting room

Over lunch

On a walk

One thing I’ve learned over time is that the highest bandwidth way of talking is face to face because you can read body language in addition to words. Video chat is okay, but an artificial and often overly formal setting. Phone is faster than text. Text drops the audio/visual/emotional signal completely. Code is precise but requires reverse engineering intent.

I personally like a walk, and then pair programming a shared screen.

I also tried using Siri to tell me the weather forcast while I was driving to the park. It asked me to auth into my phone. Then it asked me to approve location access. I guess it was secure but I never figured out what the weather forecast was.

Thankfully it didn't rain on my picnic. Some of the parents there asked me if their investors should be interested in clawd.

If you are a budget approver then your inbox and calendar are full of sales teams.

I find that experience is too distracting to concentrate on writing good code.

I especially agree with your point that scaling laws really killed open research. That's a shame and I personally think we could benefit from more research.

I originally didn't like calling them scaling laws.

In addition to the law part seeming a bit much, I've found that researchers often overemphasize the scale part. If scaling is predictable, then you don't need to do most experiments at very large scale. However, that doesn't seem to stop researchers from starting there.

Once you find something good, and you understand how it scales, then you can pour system resources into it. So I originally thought it would encourage research. I find it sad that it seems to have had the opposite effect.

From a technology or engineering perspective, I have no idea how to work with Apple.

What high quality data sources are not already tapped?

Where does the next 1000x flops come from?

Over time we seem to have a tendency to build models that are well matched to our machines

That’s inference code, but where is the high perf web server?