Hacker News: cdadminhttps://news.ycombinator.com/user?id=cdadminHacker News RSShttps://hnrss.org/hnrss v2.1.1Sun, 27 Sep 2026 02:44:57 +0000<![CDATA[New comment by cdadmin in "Facebook employee responds on robots.txt controversy"]]>Please post contact information for bots wanting to be whitelisted. (Either here or in the robots.txt file itself).

For example, our ChangeDetection.com site has a couple hundred users monitoring pages on facebook. (User-Agent: ChangeDetection). We always honor robots.txt so if we are not whitelisted all these monitors will be disabled.

Edit: looks like you already have posted contact info in robots.txt

]]>Fri, 18 Jun 2010 16:29:48 +0000https://news.ycombinator.com/item?id=1442733cdadminhttps://news.ycombinator.com/item?id=1442733https://news.ycombinator.com/item?id=1442733