Hacker News: agajews

New comment by agajews in "A General Goal-Conditioned Minecraft Model"

agajews — Wed, 15 Jul 2026 21:06:16 +0000

Pretty different--obviously a lot less consistent today, but capable of a lot more diverse kinds of behavior. As the models get bigger, they'll be easier to prompt, and you could imagine a game designer writing a long prompt for different kinds of NPCs. (E.g. GPT-3 was pretty difficult to get to do a particular thing, although it was very general, but over the years the instruction tuning has gotten a lot better and now it's very easy to ask questions to ChatGPT)

New comment by agajews in "A General Goal-Conditioned Minecraft Model"

agajews — Wed, 15 Jul 2026 21:03:44 +0000

Awesome! We should be a lot better than ordinary LLMs, especially at tasks that require making a lot of decisions in real-time.

New comment by agajews in "A General Goal-Conditioned Minecraft Model"

agajews — Wed, 15 Jul 2026 21:02:25 +0000

Probably not the current model, but one of the benefits of doing internet-scale pretraining is that the model has seen a lot of mods already! We think the bigger models will be able to handle some more custom Minecraft worlds.

New comment by agajews in "A General Goal-Conditioned Minecraft Model"

agajews — Wed, 15 Jul 2026 18:20:32 +0000

The goal conditioning is just a training objective! You could combine it with tool use, web searches, etc. and still train end-to-end on goal conditioning. (One way of thinking about it is goal conditioning defines a good distribution of tasks, and lends itself well to pretraining with an unlabelled video dataset.)

New comment by agajews in "A General Goal-Conditioned Minecraft Model"

agajews — Wed, 15 Jul 2026 18:17:51 +0000

you don't need to pick good images manually! it's similar to next-token prediction, many prediction tasks aren't especially interesting, but there are enough hard ones that the model can spend the most time learning from those. the simplest thing scaled up works very well.

New comment by agajews in "A General Goal-Conditioned Minecraft Model"

agajews — Wed, 15 Jul 2026 17:50:14 +0000

Not yet, but we'll add language as a modality to the larger models! The models are trained end-to-end on video data, so we'll need datasets that mix video and language, e.g. transcripts of game streams. When the models are scaled up to cross-videogames and robots there will definitely be a bunch of language data.

New comment by agajews in "A General Goal-Conditioned Minecraft Model"

agajews — Wed, 15 Jul 2026 17:20:59 +0000

Very much inspired by those papers! One of the things that's interesting about our model is it's goal-conditioned, so it can do any task at inference time without training on it. We had a lot of fun making eval environments after we trained the model trying to find interesting things it can do, and that was all after we trained the model. More like prompting an LLM.

(Versus Dreamer, which needs to be trained on a hand-written reward function for each task that you want to do.)

New comment by agajews in "A General Goal-Conditioned Minecraft Model"

agajews — Wed, 15 Jul 2026 17:18:07 +0000

Hey everyone! I'm Alex, one of the founders of Pantograph. We've spent the last six months building a pretty smart Minecraft model, coming soon to a server near you!

We trained it on about 500k hours of Minecraft videos, and it learned how to fight creepers, build walls and other structures, and explore to find visual goals.

We're considering putting up a public API for larger models like this one, let us know if you'd like to be able to put Pan in your own server :)

What's most interesting about the model isn't the performance that it gets in Minecraft, but how general the method is. When we scale it up, it should be able to act in any kind of video game, as well as robots in the real world (which are really just another video game).

A General Goal-Conditioned Minecraft Model

agajews — Wed, 15 Jul 2026 16:35:09 +0000

Article URL: https://pantograph.com/journal/pan-1

Comments URL: https://news.ycombinator.com/item?id=48923412

Points: 33

# Comments: 15

Pantograph: Building a preschool for robots

agajews — Tue, 23 Dec 2025 19:59:19 +0000

Article URL: https://pantograph.com/blog/building-a-preschool-for-robots.html

Comments URL: https://news.ycombinator.com/item?id=46368827

Points: 44

# Comments: 9

New comment by agajews in "Show HN: San Francisco Compute – 512 H100s at <$2/hr for research and startups"

agajews — Sun, 30 Jul 2023 22:45:59 +0000

Ah the hardware isn’t gonna be in SF (not the cheapest datacenter space)

But I do think a lot of our customers will be out here —- SF is still probably the best place to do startups. We just have so many more people doing hard technical stuff here. Literally every single place I’ve lived in SF there’s been another startup living upstairs or downstairs

Good idea to host some in person events!

New comment by agajews in "Show HN: San Francisco Compute – 512 H100s at <$2/hr for research and startups"

agajews — Sun, 30 Jul 2023 22:15:35 +0000

Yeah we aren’t going to let anyone book the whole thing for years. If we ever have to make the choice, we’ll choose the startups over the big companies.

New comment by agajews in "Show HN: San Francisco Compute – 512 H100s at <$2/hr for research and startups"

agajews — Sun, 30 Jul 2023 22:10:05 +0000

Ah that’s only if you pay for 3 years of compute upfront. Most startups, especially the small ones, really can’t afford that

New comment by agajews in "Show HN: San Francisco Compute – 512 H100s at <$2/hr for research and startups"

agajews — Sun, 30 Jul 2023 19:46:26 +0000

Yeah it’s pretty hard to find a big block of GPUs that you can use for a short time, esp if you need infiniband for multinode training. Lambda I think needs a min reservation of 6-12 months if you want IB.

New comment by agajews in "Metaphor Systems: A search engine based on generative AI"

agajews — Thu, 10 Nov 2022 20:24:12 +0000

Yeah exactly. That's why you can't really do it with a language model like GPT-3, you have to bake into the architecture the concept of a "link" as a first-class object.

New comment by agajews in "Metaphor Systems: A search engine based on generative AI"

agajews — Thu, 10 Nov 2022 19:55:23 +0000

Hey, thanks for posting!

We actually have an architecture that lets us expand the index without doing any retraining, so we can add/update pages pretty much for free.

New comment by agajews in "Metaphor Systems: A search engine based on generative AI"

agajews — Thu, 10 Nov 2022 19:53:42 +0000

Hey everyone! Metaphor team here.

We launched Metaphor earlier this morning! It's a search engine based on the same sorts of generative modeling ideas behind Stable Diffusion, GPT-3, etc. It's trained to predict the next link (similar to how GPT-3 predicts the next word).

After GPT-3 came out we started thinking about how pretraining (for large language models) and indexing (for search engines) feel pretty similar. In both you have some code that's looking at all the text on the internet and trying to compress it into a better representation. GPT-3 itself isn't a search engine, but it got us thinking, what would it look like to have something GPT-3-shaped, but able to search the web?

This new self-supervised objective, next link prediction, is what we came up with. (It's got to be self-supervised so that you have basically infinite training data – that's what makes generative models so good.) Then it took us about 8 months of iterating on model architectures to get something that works well.

And now you all can play with it! Very excited to see what sorts of interesting prompts you can come up with.