Now there are reasons you might want to know about the training data when you wouldn't care about the authoring process used by a traditional open-source process. And these get at the reason LLM's are different than traditional software and so maybe our existing definitions of what 'open-source' means aren't a good match for LLM's. Of course there is software associated with LLM's (beyond the weight) -- defining the structure of the particular neural net those weights fit into. In every open-weights model that I'm aware of that software is open source (though trivial).

To a useful level of accuracy we can certainly simulate water. And we can do the same for a single proton for some definitions of useful (but not other definitions).

At least if the metaphor is about coding as a means to creating usefully functional code as efficiently as possible. Careful coding by hand may eventually be a hobby activity.

Personally, while I do get some satisfaction in coding by hand it was always the production of something useful that I found most rewarding. I was never someone who wrote code for a hobby. With LLM's I'm more productive. And I find that very satisfying.

Not saying LLM's are conscious. Just that much of our amazement about their behavior seems to say more about us realizing the things we can do are not magic, rather than them being so amazing.

Now granted, if the boat salesmen were pushing hard on the idea that the boat would fly and even put little wings on the side and I bought the boat I might get really angry when I found out that it didn't fly. And I might angrily storm into the salesroom yelling about how the design is defective. But if someone pointed out 'hey, it's a boat perhaps you should stick to sailing around in it and stop getting your undies in a bundle about it not flying' the correct response is probably to take a closer look, ignore the salesmen, and cruise around the lake. LLM's are quite handy at some things and have some weird limits. Learn the limits, enjoy your time at sea.

FWIW while neither model included your product in it's initial response, when I followed up with "what about open-source" both did another search and Claude's response included your tool....