However day after day it just gets blasted with requests for deep pages. I was quite alarmed when I saw a 100x increase in the daily active user numbers which relied on session length, only to realize they were all bots.

Naively I too initially resorted to blocking user agents(Meta is thankfully nice enough to identify themselves, not nice enough to stop blasting 50k requests a day however), IP ranges from cloud providers and various browser fingerprints that I found connected to suspicious traffic.

However the battle seems unwinnable at the moment, outside of gating all content behind auth which I don't want to do. We have around 500k user generated content pages and I want those to remain publicly available.

I would be happy to provide our data to any one of these scrapers and I even added a message asking them to contact us if they want access to our data whenever I return a 403 response, however nobody has reached out.

Another campaign that someone is constantly running is daily checks for 100s of possible secret/config paths in hopes of finding an exposed private variable, these i've just blocked even though they would return a 404.

I still haven't found a way to deal with rotating residential IPs however, and most likely never will.

My current approach is to just run a 24 hour scan of all requests with codex and update my next.js proxy with more IP ranges, browser fingerprints and anything else that won't affect a real person.

Has anyone managed to come up with a way to stop this onslaught of crawlers and scrapers?

Which is a shame as real insights are buried inside some of these articles, which if the author bothered to write in his own words could have reached an audience that would have appreciated them.

Writing is one of the areas where I want no LLM involvement.

My previous solution included yanking the display cable out every time I wanted to close the lid and then plugging it back in.

After looking for an easy solution, I tried https://github.com/pirj/noclamshell but I didn't like the approach it took where it basically polled every 2 seconds and reacted to the lid state at the time of the poll. This + using a stale state reading led to a bug where it activated sleep mode with the lid open. I also did not want to install an app as I see this as a lightweight utility your turn on and forget, not something taking up menu bar space.

The way it works is by listening to native macOS state for power, external displays and the lid position, resulting in a noop if no power of external display is connected or sending the external display to sleep if all conditions are met. The cable still continues to charge the battery even with the lid closed.

Comments URL: https://news.ycombinator.com/item?id=48885362

Points: 3

# Comments: 0

Perhaps the next target for a 100x improvement

The gap between a pet project and great software is still very wide and I have a hard time believing that it will ever be bridged.

I don't see how a solved problem even before ai is the thing that won't be replaced first. I struggle to believe a personal project requires complex infra

Seems to me the way to go with these apps that keep pushing daily updates.

At the same time I was going through it trying to implement a handful of Rive animations in our app built with React Native, with not so stellar results and a taste in my mouth so bad that it inspired me to write about it.

Mostly to discourage other small teams(with a solo dev like us) saving their roadmaps in process and to share how I intend to tackle animations in the future.

Comments URL: https://news.ycombinator.com/item?id=46268125

Points: 2

# Comments: 1

Dreaming big with hopes of becoming a household name similar to imdb or letterboxd.

Looking at your site I think you would like to hear that when rating games you can mark a game as 'attended', 'watched' or other categories to show others how you experienced the game or just for your own personal log.

Other than that we are light on classic sports stats and our stats are more focused on the user ratings of the games rather than the game stats.

https://rategame.io/ and on the app stores(I really recommend the app if you want to check it out, especially on a phone)

We've expanded on the concept with rating stadiums, creating lists, voting for player of the game and more as we are trying to become the letterboxd for sports.

[1]: https://extensions.dev/extensions/meilisearch/firestore-meil...

As you get closer to it, the post office stands out and the tower is the one thing that always catches my attention in a sea of blandness (not to complain as these old apartment blocks are nice to live in, I live in one myself).

The windows become orange and it feels like a villain is plotting his world domination somewhere up there. The type of villain whose plans never succeed and his tower is beginning to crumble.

This will be a great example to see how the library has handled the same problem.