https://news.ycombinator.com/item?id=36993739

To massively (over)simplify the argument to its essence (and ignore other important points): the publisher goes through the trouble and expense of creating the content The publisher then allows its content to be copied by a search engine only because being shown in search results gets it traffic back. The traffic it gets in return has value, and the publisher is happy for this arrangement to continue as long as the value of the traffic is more than the cost of producing and serving the content.

Brave offering a "license", for its own financial benefit, to "allow" others to use the content for LLM training gives zero benefit to the original publisher. This is why I use words like "sleazy" to describe Brave's position.

This argument applies to Google and Microsoft. Right now both are failing at citing sources in their generative AI search results. That is terrible and I hope it's fixed soon, as otherwise they're being sleazy scrapers as much as Brave is.

Finally, I wholeheartedly disagree they what Brave is doing is for the "greater good". The fact they charge extra for the "license" to use the content for LLM training shows that.

I share your concern about Google having this much power, and I'd add that Microsoft Bing is equally bad but gets away with it because they're smaller. Still, the final decision about which search engine indexes a website is purely the publisher's.

Also, these search crawls by the browser do not identify themselves beyond the Brave standard UA header, namely a plain Chrome user-agent string.

Even worse, they state that Brave search won't index a page only if other search engines are not allowed to index it. It is morally not their right to make that call. A publisher should have full control to discriminate which search engine indexes the website's content. That's the very heart of why the Robots Exclusion Protocol exists, and Brave is brazenly ignoring it.

Even worse than that, the Brave search API allows you (for an extra fee) to get the content with a "license" to use the content for AI training? Who allowed them the right to distribute the content that way?

I wrote about all this here:

https://searchengineland.com/crawlers-search-engines-generat...

and more references elsewhere in this thread:

https://news.ycombinator.com/item?id=36989129

Amusingly, while I was writing my article, this got posted to their forums, asking about how to block their crawler:

https://community.brave.com/t/stop-website-being-shown-in-br...

No reply so far.

Google's official position was published on 8 February here:

https://developers.google.com/search/blog/2023/02/google-sea...

It's a much more nuanced position that can be summarized as "make sure you create good content, however you create it". A focus on quality, not process, is reasonable.

Disclosure: ex-Googler in search.

Comments URL: https://news.ycombinator.com/item?id=21983567

Points: 4

# Comments: 0

Another place this idea comes up is a search engine index. If the algo doesn't find, for a given query, documents in the index it should have (falsely classified as not matching the query), it will have lower recall.

https://www.nature.com/scitable/topicpage/mitosis-and-cell-d...

The faint lines are the cell walls and the bright spots in the middle would be the DNA. I can believe this is what they're going for with a bit of squinting.

1. Add other browse extensions, and you'll see a big difference between their effects. Defaults matter a lot in this space.

2. Compare mobile vs desktop. Getting mobile emulation to be good enough is a bit of work, but worth it IMO.

Based on internal usage, the typical web page will load 35-45% faster with uBlock Origin installed.

My email address is my profile if you want to compare notes or whatnot.

https://news.ycombinator.com/item?id=20394661

The GDPR talks about online identifiers, of which cookies, IP address and fingerprints are examples. If you read any regulator's guidance carefully, you'll see they talk about "cookies and similar technologies", with just "cookies" being used alone for brevity.

To rephrase tracking of any kind is the issue, not cookies. Don't mistake the implementation for the activity.

Disclosure: Founder of a non-tracking web analytics service because of this exact issue.

The privacy policy is very not suited for this service. The most important point is that you're based in Germany based on the address in the policy, but there isn't a single mention of the GDPR. That and the ePrivacy Directive are what count for you the most. My recommendation is don't use a free policy generator and get proper advice. I appreciate this isn't something commonly seen as a launch blocker, but it's important to sort it out properly.

Find your German state data protection authority, and invariably you'll find they have great guidance.

It’s easy to see why quote I gave says what it says with this context.

Also, if you’re worries, talk to your lawyer.

“Natural persons may be associated with online identifiers provided by their devices, applications, tools and protocols, such as internet protocol addresses, cookie identifiers or other identifiers such as radio frequency identification tags. This may leave traces which, in particular when combined with unique identifiers and other information received by the servers, may be used to create profiles of the natural persons and identify them.”

Source: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A...

I'm curious why you chose to host the data yourself instead of giving customers the data immediately at the point of collection. That's the path we chose for Blockmetry as it genuinely required to be a non-tracking web analytics service and makes it impossible to profile users. Any service that hosts its data would still be open to being untrusted on the "no tracking no profiling" argument.

Thanks, Pierre

PS - YC Startup School founders: ping me via the forums and get an extended-period free trial.

BTW, are you the same Peter Hartree on this Segment thread? https://community.segment.com/t/1889n1/how-common-is-client-... It would appear we've crossed paths before on this topic. Please do email me if you want to talk properly. That Segment thread has my email.