I don't like the analogy, it supposed that AI generated code will always be badly organized underneath if it is automatic, however, as time goes, I find the code better and better..

On another note, I'm working on a (big) PEP for python, and looking for maintainers and sponsors (as per python's rules). Do someone know where I could find such amazing persons? (I already have the code, works in cython,graalVM and pypy, and it's kind of a big change but I believe it will help a lot the language)

This was to be expected, llms are trained on a very large corpus of data, they 'look' like creative from our point of view, as humans don't have that much knowledge, so, if the LLM proposes something that a human does not know, it looks like creative. But they stay neural networks after all, so very good in interpolation, very bad at extrapolation. Building new knowledge is extrapolation if it's truly outside the convex hull how the dataset.

I usually write code and ideas that are at the border of the convex hull, I haven't seen yet an llm being able to propose a really innovative solution. All the time they reuse building blocks that are already there. And I really tried pushing by making them creating proofs.

If I have an insight on some problem and I ask for the proof for it, it will usually be able to generate the proof, but that proof is never creative, and often it will use novel ideas only if I explain the novel idea. It will not come up with it alone. (And sometimes it's very hard to even explain the ideas, as it tends to return to knew waters)

Its only a data point, but I haven't seen a paper that handle this issue yet, and I don't expect them to magically become good at extrapolation due to the theory behind neural networks.

However, at the moment I consider that they stay in the 'convex hull' of their training set + a provided context, and I don't see that much research that made real improvements to the situation.

Most of programming is reusing existing ideas in new shapes to solve new problems, but all the building blocks are there in the training set. Or new blocks can (easily) be derived from existing ones.

Math is different, it requires quite a bit of creativity, it's not just 'reuse all existing blocks'.

For the moment LLMs are good at discovering things that we overlooked in maths, or apply cleverly existing math blocks to make new results, but making a new theory that is really useful is out of reach for the moment in my opinion.

I was looking at your website and wondering if there is a way to have access to the course material/videos?

In particular: Spring 2025 @ UR : CSC 209/409 Seminar on Artificial General Intelligence

Fall 2024 @ UR : CSC 277/477 End-to-End Deep Learning

Spring 2023 @ UR : CSC 266/466 Frontiers in Deep Learning

Spring 2022 @ Cornell Tech : CS 5787 – Deep Learning

Fall 2021 @ RIT : IMGS 684 – Deep Learning for Vision

It would be lovely to parse which datasets/benchmarks were used in the comparisons and select papers by dataset!

In many fields the datasets vary greatly depending on the subfield and its very difficult to find what other benchmarks could be used.

The other main issue that I see, is that even if there is a formally verified specification, at the moment, LLMs will not respect it perfectly. As long as LLMs are not able to non-deterministically follow a spec, the technology is not good enough.

A part from that, imo, in this age we should focus more on the mathematical aspect of computations, and I think we need to develop novel theories that take into account the non deterministic nature of LLMs in the process. I'm not sure this will ever work by merely extending current practices, as software design practices are extremely poorly defined from an engineer point of view. Just extending them by including randomeness does not seem a good idea.

If someone else struggle on this step, I added in vLLM args: --served-model-name "Qwen2.5-Coder-14B-Instruct" --served-model-name "default"

So default becomes an alias.

I didn't yet test Forge, I was just happy that it worked at the moment ;)

Massively Parallel Processors: A Hands-on Approach is not really good in my opinion, many small mistakes and confusing sentences (even when you know cuda).

CUDA by Example: An Introduction to General-Purpose GPU Programming is too simple and abstract too much the architecture.

Next year I'm planning to start writing a cuda book that starts by engineering the hardware, and goes up to the optimization part on that harware (which is basically a nvidia card) including all the main algorithms (except for graphs).

I'm already teaching the course in this way at uni, and it is quite successful among students.