Similar to the one SQLite had: https://www2.sqlite.org/forum/forumpost/7d3eb059f81ff694?t=h

Each IP only makes ~1 request though so easy to detect after the fact.

I guess they will run out of IPs at some point so maybe if I had logged each one forever and shown a challenge only to them, it would have fixed it eventually. Just depends how big their pool of IPs is.

I did try removing some of the links without success. I guess once they have them they just keep checking.

It's also not always easy to do. I run a small wiki which is fairly optimised, nearly every page manages at least ~3k rps on a small VPS. The only exception is the diff page which is ~150 rps. Optimising that while still giving good output isn't that easy, but the wiki doesn't have many users so that would be fine if it wasn't for the AI bots.

The AI bots ignore robots.txt and were initially hitting the site with ~1k rps crawling every combination. Even that would be manageable as there's currently ~150,000 combinations, except they kept re-crawling the whole lot each day. The server could manage it but it was a massive waste of resources.

They were using residential IPs and only sending 1 request from each IP making it impossible to block. In the end I gave up and put a Cloudflare challenge in front of it. I don't want to use Cloudflare but the alternative is forcing users to login to view diffs or remove them entirely.

I do think it is still very true for tools though. It's nearly always worth getting decent ones, they nearly give better results or are easier to use and last so much longer.

I have a site which is currently being hit (over 10k requests today) and it looks like scrapers as every URL is different. If it was a DDoS, they would target costly pages like my search not every single URL.

SQLite had the same thing: https://sqlite.org/forum/forumpost/7d3eb059f81ff694 As have a few other open source repositories. It looks like badly written crawlers trying to crawl sites as fast as possible.

The whole point of them was to give the US influence while improving US security. Given Europe can't trust Trump will come to their aid, they won't give the US as much influence over Europe.

Having said that, it doesn't seem very active at the moment which isn't a good sign. Granted, I have no idea how active the subreddit is to compare.

The only reason the privacy shield agreement was thrown out was due to lack of safe guards from US intelligence.

Even without the privacy shield, US companies would still be able to store EU data in a country with an adequacy decision if it wasn't for the CLOUD act. This seems more to do with US law wanting access to EU data.

It looks like the court decided SCCs were not sufficient as Cloudflare is subject to US surveillance laws so they wouldn't be able to provide adequate guarantees.

[1]: https://www.fastmail.help/hc/en-us/articles/1500000277382-Ac...