I’ve been using diffusion Gemma and it is very fast on GPUs in output token/sec.

In the diffusion Gemma whitepaper, they say they could have done better with more time and compute.

Even with those caveats, it is very uses-able as a local model.

Shouldn't that be 0 innovation tokens though?

Instead I like “only work on impossible problems”

Most of them turn out to be impossible, but some of them turn out to be possible.

I’ve never met anyone who could pick 3 and be confident in getting even one right. Tokens are a terrible analogy for innovation or research.

In hindsight I’ve had to sift through hundreds or more to fine one that worked.

I thought this post was helpful when I first started thinking about startups.

After more time, I think boring tech isn’t worth thinking about.

I wonder how much it would cost to vibe code the whole thing from scatch?

I wonder how much better models need to get before such a thing wouldn't look like code vomit?

That means that 10 years from now I don’t want think about making sure the hosting server is up.

I also want it to have a standard format for bibliography, DOI, and authors.

I agree it isn’t much, but it’s more than I get from a regular web hosting service and it is a standard format for papers so I don’t think it makes sense for every author to roll their own.

I wouldn't want my google drive to start telling me my paper was too sloppy. I just want a link.

There is a lot of follow on work that explains what happens as you change them, e.g. Scaling Laws for Transfer - https://arxiv.org/pdf/2102.01293

I think it’s fortunate that transfer works in a similar way.

Common crawl (and Reddit, stack overflow, etc but not 4chan) was much easier to get access to at the time than using mechanical Turk.

There is certainly room for more work. There were many papers on scaling laws in NeurIPS this year.

I started to believe it after we (Joel Hestness in particular) reproduced it in so many experiments in “scaling is predictable empirically”.

The OpenAI work replicated it in a completely different environment, and at that point I was sure it was real.

Sometimes people ask me why I was so surprised by it. Prior work like Banko and Brill and the unreasonable effectiveness of data argued for more data. ML theory had similar models for toy problems, eg coin flips.

At the time I thought deep learning was supposed to be complex. Speech and language datasets seemed much more complex than toy problems. Optimization of deep transformers was complex.

The idea that it was possible for the whole thing to be governed by a 3 term equation seemed too simple. The implication was that it was simple to manufacture intelligence.

Ten years later, I still think it is still the most interesting observation I have seen. We are still learning what it looks like to live in a world where it is possible to manufacture intelligence.

I tried basic elegoo and bambu printers.

He can’t read very well but he likes dragging shapes around on a tablet.

He would ask me to find shapes using the search engines then he mixes them together or reshapes them.

I would add them to his history.

This is why I was surprised to hear about 3D printed guns. I was quite sure there wasn’t anything like that in the history.

It was a good discussion topic about why adults get so bothered by things that look like guns.