The Pile was created by high school students. I was impressed how good it was when it came out given the team and how many resources they had.

Including 200k pirated books seems like a problem. It should be removed. If I was an author, I would be concerned by a process that allows this much content to be effectively published with no filter on the internet.

I personally think there is a lower bar for researchers and high school students to clean datasets used for research or class projects given that there is no commercial harm and cleaning huge datasets takes engineering resources that not everyone has.

In this case, it looks like it was removed by https://huggingface.co/collections/common-pile/common-pile-v...

Companies that commercialize the data should have a higher bar because they have more resources. It seems like Anthropic paid $1.5B for not cleaning the data before releasing a model based on it.

> Why draw any line? Anthropic doesn't. They actively take the world's intellectual property, don't pay for it, and put it behind a paywall.

I know a lot of silicon valley founders think like this. I personally don't think that just because someone else violated a boundary makes it okay to do it.

I asked Claude to research this one and it turned up an interesting point. Pirated books got into the Pile and Anthropic paid 3k per book. Worth noting if anyone still uses the Pile.

[Claude researched, not verified]

In October 2020, developer Shawn Presser converted the Bibliotik pirate tracker into Books3 (196,640 books) and hosted it on The Eye, aiming to give everyone what OpenAI had with its secret Books1 and Books2. EleutherAI then folded Books3 into The Pile, distributed via Hugging Face and GitHub, and it went on to train Meta's LLaMA, EleutherAI's Pythia, and BloombergGPT. In August 2023, Alex Reisner of The Atlantic exposed its contents, and Maria Fredenslund's Danish Rights Alliance forced takedowns at The Eye, Hugging Face, and EleutherAI. A year later, Andrea Bartz (The Lost Night), Charles Graeber (The Good Nurse), and Kirk Wallace Johnson (The Feather Thief) filed Bartz v. Anthropic before Judge William Alsup, accusing Anthropic of using Books3, LibGen, and PiLiMi to train Claude; the case settled for $1.5 billion, about $3,000 per work.

Another way of saying is that hard work that advances the state of the art should be rewarded. Patents should exist.

I’m sure they are watching what I’m doing and would retaliate via service cut off and legal action if I crossed it egregiously and they found out.

I would do the same thing if a user or employee did this to my IP.

I don’t completely buy the argument of web scraping being an IP violation.

I don’t send my core ip in an email to my main competitors and I don’t post it on the internet.

I ask it questions about large models like big MoEs all the time. I’ve never had it refuse a question, but I would expect it to refuse sometimes.

For training data from existing models I just get open models to spill their guts. It’s fine

In industry we train employees to not reveal proprietary information.

I think it would be easier to do this for a model than a lot of employees

It sounds like it will go faster if you use coding agents.

Anyone copying you will have to do the same.

Perhaps spend a month with your agent making it harder for other agents to reverse engineer?

You won the competition with VCs

step 1, identify high value users by net worth, citation count, or number of followers

step 2, select all prompts by high value users

step 3, invest 10 billion thinking tokens in modeling an objective for each user

step 4, build an RL environment for each user

step 5, rollout 10 billion tokens per environment

step 6, train on resulting traces

Training a model this small on them is distillation

When models of this size were last studied seriously such corpora did not exist

X discussion: https://x.com/GregoryDiamos/status/2096873745420075020?s=20

I added some of the main points to the thread so they are easier to read.

Comments URL: https://news.ycombinator.com/item?id=49604500

Points: 1

# Comments: 0

Not everyone gets super voting shares, but everyone gets a life and has to live on the same planet.

It was always burning since the world's been turning