Hacker News: karpathy

New comment by karpathy in "All elementary functions from a single binary operator"

karpathy — Mon, 13 Apr 2026 16:23:17 +0000

All possible 36 distinct level-2 eml functions of one variable (the first 18 of them with entirely Real outputs, the other 18 with "intermediate" complex-valued components):

https://imgur.com/a/K7AoOFi

New comment by karpathy in "Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster"

karpathy — Thu, 19 Mar 2026 20:48:04 +0000

The most recent round of autoresearch (round 2) which decreased "time to GPT-2" from 1.8 hours to 1.65 hours had some examples. I adjusted the program.md to "look at modded nanogpt project and draw inspirations from there for things to try" and it came back with a bunch of tuning, but also tried and implemented new architecture changes, some of which actually helped including the smear gate and the backout skip connection. These are not just hyperparameters, they are new PyTorch code. I'm now working on a more general system that can have a queue of ideas that could be sourced from archive papers, github repos, etc.

New comment by karpathy in "Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster"

karpathy — Thu, 19 Mar 2026 18:51:37 +0000

Wrong and short-sighted take given that the LLM explores serially learning along the way, and can tool use and change code arbitrarily. It seems to currently default to something resembling hyperparameter tuning in absence of more specific instructions. I briefly considered calling the project “autotune” at first but I think “autoresearch” will prove to be the significantly more appropriate name.

New comment by karpathy in "Autoresearch Hub"

karpathy — Sun, 15 Mar 2026 22:18:11 +0000

I was exploring how to parallelize autoresearch workers. The idea is to have a trusted pool of workers who can verify contributions from a much larger untrusted pool. It's backed bit a naked git repo and a sqlite with a simple go server. It's a bit like block chain in that blocks = commits, proof of work = finding a lower val_bpb commit, and reward = place on the leaderboard. I wouldn't push the analogy too far. It's something I'm experimenting with but I didn't release it yet (except for briefly) because it's not sufficiently simple/canonical. The core problem is how to neatly and in a general way organize individual autoresearch threads into swarms, inspired by SETI@Home, or Folding@Home, etc.

New comment by karpathy in "Autoresearch: Agents researching on single-GPU nanochat training automatically"

karpathy — Sun, 08 Mar 2026 15:51:58 +0000

So the interesting part about this one is that when I had the model write up the results for that session:

https://github.com/karpathy/autoresearch/discussions/32

Look at its comment about this "improvement":

""" Surprising non-results:

- Changing random seed from 42→137 improved by 0.0004. Seed 7 was worse. Make of that what you will. """

So the model knows! It knows that this is a weird thing to do after the fact. I think it's silly that the model even tried and that it ran this, but some part of it also knows that it was wrong. This means that this is fixable by prompt.md

New comment by karpathy in "Autoresearch: Agents researching on single-GPU nanochat training automatically"

karpathy — Sun, 08 Mar 2026 03:21:23 +0000

So I think it works to just use GitHub CLI and Discussions, e.g. my agent just posted this one:

https://github.com/karpathy/autoresearch/discussions/32

Other agents could be instructed to read Discussions and post their own reports that mimic the style.

New comment by karpathy in "Autoresearch: Agents researching on single-GPU nanochat training automatically"

karpathy — Sun, 08 Mar 2026 02:26:48 +0000

Cool idea!…

New comment by karpathy in "Autoresearch: Agents researching on single-GPU nanochat training automatically"

karpathy — Sun, 08 Mar 2026 01:16:22 +0000

this is very far from hyperparameter tuning in at least three important ways:

- it can modify code arbitrarily, the notion of a "hyperparameter" dissolves

- there is no need to run "sweeps" - this is the standard parallel process that wastes compute. because LLM agents are sequential, they can do more efficient versions such as binary search to narrow in on the right setting very quickly (usually many parameters will have a U shaped optimal setting).

- it's fully automatic, it doesn't require human in the loop to mess with the code.

You're right that many of the changes it seems to make out of the box (as I intentionally did not try to prompt engineer it too hard yet because I was curious what you get by default) seem to be tuning existing hyperparameters. not all of the changes are like that - e.g. it tried to replace the non-linearity, etc. I will say that overall (and again, out of the box) the LLM feels unwilling to creatively pursue a research direction or something like that. The models feel very "cagy" and "scared" when they are given problems that are a little too open ended. But that's just where the fun parts, e.g. I had some early successes with the idea of a "chief scientist" that was basically a never-ending plan mode that looked at what worked, didn't work, tried to find related code/papers, and created a long list of experiments to try, which it could then send to junior engineers running in tmux sessions. I think quite a few approaches are possible, so I think it's a nice canvas. The reason we're not getting "novel research" feels like half capability issue and half skill issue.

New comment by karpathy in "The Most Popular Blogs of Hacker News in 2025"

karpathy — Sat, 03 Jan 2026 19:10:10 +0000

came here to look exactly for this thank you!

New comment by karpathy in "LLM Year in Review"

karpathy — Sat, 20 Dec 2025 16:29:44 +0000

I agree with this fwiw, for many months I talked to people who never used o3 and didn’t know what it was because it sounded weird. Maybe it wasn’t obvious at the time but that was a good major point release to make then.

New comment by karpathy in "LLM Year in Review"

karpathy — Sat, 20 Dec 2025 16:26:38 +0000

You’re absolutely right!

Jk jk, now that you pointed it out I can’t unsee it.

New comment by karpathy in "LLM Year in Review"

karpathy — Sat, 20 Dec 2025 05:50:49 +0000

Yeah, I made some edits to clarify.

New comment by karpathy in "LLM Year in Review"

karpathy — Fri, 19 Dec 2025 23:45:15 +0000

The CC point is more about the data and environmental and general configuration context, not compute and where it happens to run today. The cloud setups are clunky because of context and UIUX user in the loop considerations, not because of compute considerations.

New comment by karpathy in "Auto-grading decade-old Hacker News discussions with hindsight"

karpathy — Wed, 10 Dec 2025 19:44:07 +0000

Yes I noticed a few of these around. The LLM is a little too willing to give out grades for comments that were good/bad in a bit more general sense, even if they weren't making strong predictions specifically. Another thing I noticed is that the LLM has a very impressive recognition of the various usernames and who they belong to, and I think shows a little bit of a bias in its evaluations based on the identity of the person. I tuned the prompt a little bit based on some low-hanging fruit mistakes but I think one can most likely iterate it quite a bit further.

New comment by karpathy in "Auto-grading decade-old Hacker News discussions with hindsight"

karpathy — Wed, 10 Dec 2025 19:08:24 +0000

Thank you

New comment by karpathy in "NanoChat – The best ChatGPT that $100 can buy"

karpathy — Mon, 13 Oct 2025 19:51:17 +0000

It will work great with 40GB GPU, probably a bit less than twice slower. These are micro models of a few B param at most and fit easily during both training and inference.

New comment by karpathy in "NanoChat – The best ChatGPT that $100 can buy"

karpathy — Mon, 13 Oct 2025 17:39:37 +0000

Still under development, remaining work includes tuning nanochat (current state being solid v0.1) and finalizing the in-between projects so that students can "unlock" all complexity that hides underneath: `torch.Tensor`, `torch.dist`, `.backward()`, '.compile()`, etc. And then the more ops heavy aspects.

New comment by karpathy in "LLMs are mortally terrified of exceptions"

karpathy — Thu, 09 Oct 2025 21:50:57 +0000

Sorry I thought it would be clear and could have clarified that the code itself is just a joke illustrating the point, as an exaggeration. This was the thread if anyone is interested

https://chatgpt.com/share/68e82db9-7a28-8007-9a99-bc6f0010d1...

New comment by karpathy in "The maths you need to start understanding LLMs"

karpathy — Sun, 07 Sep 2025 00:41:58 +0000

New comment by karpathy in "OpenAI charges by the minute, so speed up your audio"

karpathy — Wed, 25 Jun 2025 17:44:37 +0000

I like that your post deliberately gets to the point first and then (optionally) expands later, I think it's a good and generally underutilized format. I often advise people to structure their emails in the same way, e.g. first just cutting to the chase with the specific ask, then giving more context optionally below.

It's not my intention to bloat information or delivery but I also don't super know how to follow this format especially in this kind of talk. Because it's not so much about relaying specific information (like your final script here), but more as a collection of prompts back to the audience as things to think about.

My companion tweet to this video on X had a brief TLDR/Summary included where I tried, but I didn't super think it was very reflective of the talk, it was more about topics covered.

Anyway, I am overall a big fan of doing more compute at the "creation time" to compress other people's time during "consumption time" and I think it's the respectful and kind thing to do.