Hacker News: ccgreg

New comment by ccgreg in "What happens when an LLM never sees material beyond fifth grade?"

ccgreg — Sun, 16 Aug 2026 15:56:44 +0000

Common Crawl has very little content from Reddit and 4chan -- both blocked us years ago.

New comment by ccgreg in "GLM-5.3: Frontier coding with emergent cyber capabilities"

ccgreg — Fri, 14 Aug 2026 22:09:33 +0000

Common Crawl is about 1 petabyte per year, compressed. Uncompressed is 4X.

New comment by ccgreg in "I built a 500k-domain search engine for makers in a weekend for $10"

ccgreg — Fri, 14 Aug 2026 22:07:28 +0000

Both IA and CCF are private foundations and accept donations.

New comment by ccgreg in "A year of fighting scrapers on my 1.5 million-page website"

ccgreg — Sat, 08 Aug 2026 08:05:27 +0000

> It's a shame that CCBot is caught in the cross fire, but that's life.

We're used to it. Sadly.

New comment by ccgreg in "A year of fighting scrapers on my 1.5 million-page website"

ccgreg — Sat, 08 Aug 2026 08:04:00 +0000

The author blocked CCBot even though CCBot isn't part of the high traffic problem -- apparently he trusted Cloudflare labeling us as an "AI Bot".

New comment by ccgreg in "TIME Is Serving AI Bots a Different Website, with Ads Built In"

ccgreg — Wed, 05 Aug 2026 18:34:12 +0000

That’s already a big business!

New comment by ccgreg in "TIME Is Serving AI Bots a Different Website, with Ads Built In"

ccgreg — Wed, 05 Aug 2026 15:44:33 +0000

Thanks for the heads up -- this isn't popular yet, and it requires some work to avoid polluting things like the Internet Archive Wayback Machine.

New comment by ccgreg in "PDF Trends 2026 Q2: Analysis of 20.6M PDFs from Common Crawl"

ccgreg — Tue, 04 Aug 2026 18:13:34 +0000

The post says:

> Because Common Crawl stores only the first 1 MB of each PDF

That limit became 5 MB in March 2025.

New comment by ccgreg in "Reddit Stock Collapses 23% as AI Eats Away at User Growth"

ccgreg — Sat, 01 Aug 2026 22:40:16 +0000

Appreciate you double-checking.

New comment by ccgreg in "Reddit Stock Collapses 23% as AI Eats Away at User Growth"

ccgreg — Sat, 01 Aug 2026 21:01:35 +0000

That isn't true. You're welcome to peruse our index to prove or disprove your claim.

New comment by ccgreg in "The Second Life of Sanskrit"

ccgreg — Wed, 15 Jul 2026 04:07:19 +0000

It's kind of interesting that you contradict much of what the article concludes, even though the article gives a lot of examples. Maybe your prediction will be true.

New comment by ccgreg in "An update on residential proxies and the scraper situation"

ccgreg — Sat, 11 Jul 2026 07:20:01 +0000

We do. First off we have a public parquet-format index of all of the urls we crawl every month. And then that also lives in a HDFS table that determines when we want to recrawl a page we've crawled before.

New comment by ccgreg in "An update on residential proxies and the scraper situation"

ccgreg — Sat, 11 Jul 2026 07:17:03 +0000

Appreciate your kind words! Many people have worked at Common Crawl over the years, and it's been a labor of love fueled by positive comments like yours and the large list of PhD theses helped by our public web dataset.

New comment by ccgreg in "An update on residential proxies and the scraper situation"

ccgreg — Sat, 11 Jul 2026 05:04:25 +0000

If you're referring to Common Crawl, which has existed since 2008, indeed your predictions are somewhat accurate. It's easy to opt out or limit what is collected. The crawling itself is inexpensive to us and the hosting is from the AWS Open Dataset Sponsorship Program. And there's no charge for downloading it.

New comment by ccgreg in "An update on residential proxies and the scraper situation"

ccgreg — Sat, 11 Jul 2026 05:00:13 +0000

We aren't sure if that really made a significant difference in Common Crawl's data quality. It does hurt our dataset from a humanities point of view, alas.

New comment by ccgreg in "An update on residential proxies and the scraper situation"

ccgreg — Sat, 11 Jul 2026 02:57:14 +0000

Common Crawl's archive has metadata that says when each record (html file) was crawled.

New comment by ccgreg in "An update on residential proxies and the scraper situation"

ccgreg — Sat, 11 Jul 2026 02:56:16 +0000

Common Crawl's dataset was downloaded in full 100 times in 2025.

We agree that it would be great if it was even more widely used.

New comment by ccgreg in "An update on residential proxies and the scraper situation"

ccgreg — Fri, 10 Jul 2026 22:15:17 +0000

A lot of websites want "bot defense" due to high volume scrapers, and that "bot defense" often also ends up blocking low-volume wget/curl and polite crawlers like Common Crawl's CCBot.

New comment by ccgreg in "Free full BGP feed. IPv4 and IPv6 (2020)"

ccgreg — Sun, 31 May 2026 05:30:09 +0000

Good timing, I'm about to release that dataset.

New comment by ccgreg in "Big tech's anti-labor playbook has come for Wikipedia"

ccgreg — Fri, 29 May 2026 01:08:31 +0000

Common Crawl is working hard to improve diversity in our crawl.