> [W]e shipped an optimization. Detect duplicate files by their content hash, use hardlinks instead of downloading each copy.

(Some say ZFS as well, but it's not nearly as easy to use, and its license is still not GPL-friendly.)

https://graphite.com/blog/bitkeeper-linux-story-of-git-creat...

You may find this 10-year-old thread on HN enlightening, too: https://news.ycombinator.com/item?id=11667494

Anthropic settled the case with the publishers just a few months later, leaving the question mostly unsettled still.

"[T]he test requires that we contemplate the likely result were the conduct to be condoned as a fair use — namely to steal a work you could otherwise buy (a book, millions of books) so long as you at least loosely intend to make further copies for a purportedly transformative use (writing a book review with excerpts, training LLMs, etc.), without any accountability."

See also p. 31:

"The downloaded pirated copies used to build a central library were not justified by a fair use. Every factor points against fair use. Anthropic employees said copies of works (pirated ones, too) would be retained 'forever' for 'general purpose' even after Anthropic determined they would never be used for training LLMs. A separate justification was required for each use. None is even offered here except for Anthropic’s pocketbook and convenience."

Despite this consideration, the court still found for Anthropic on the question of fair use.

There's a doctrine in Fifth Amendment law called "fruit of the poisonous tree." The general rule is that prosecutors don't get to present evidence in a criminal trial that they gained unlawfully. It's excluded. The jury never gets to see it even if it provides incontrovertible evidence of guilt. The point is to discourage law enforcement from violating the rights of the accused during the investigative process, and to obtain a warrant as the Amendment requires.

It seems to me that the same logic ought to be applied to these companies. They want to make money by building the best models they can. That's fine! They should be able to use all the source data they can legitimately obtain to feed their training process. But if they refuse to do so and resort to piracy, they mustn't be allowed to claim that they then used it fairly in the transformative process.

Remember the golden rule: if you can't be bothered to write it yourself, why should your audience be bothered to read it ourselves?

A man page source isn't a binary format, so your statement that they're "plain text" is technically correct. (The same is also true of TeX and LaTeX files, and even PostScript if you want to stretch the definition of "plain text" until it snaps.) But the renderer is groff or (legacy) troff with the `an` macro set. less(1) (or, originally, more(1)) is just the pager that consumed the former's output (if the output format is ASCII, which is one of many) and handled paging on the terminal for the convenience of the user.

In my old Sun workstation (and even early Linux desktop) days, I rarely used man(1) in the terminal because 1/terminals were usually too small and weren't usefully resizable like they are today, and 2/unadorned monospaced fonts don't look nearly as nice as properly typeset pages do. (Color terminals were just coming on the horizon, and text could only be emboldened, not italicized.) Instead, I typically used xman whenever I could. The best way I can describe xman is as if you were rendering man pages into PDFs and viewing them in Preview on the Mac. Man pages were much more comfortable to read that way.