Hacker News: Firefishy

New comment by Firefishy in "OpenStreetMap overwhelmed by bots scraping data"

Firefishy — Wed, 28 Jan 2026 19:15:53 +0000

Disclosure: I am part of the mostly volunteer run OpenStreetMap ops team.

Technically we able to block and restrict the scrapers after the initial request from an IP. We've seen 400,000 IPs in the last 24 hours. Each IP only does a few requests. Most are not very good at faking browsers, but they are getting better. (HTTP/1.1 vs HTTP/2, obviously faked headers etc)

The problem has been going on for over a year now. It isn't going away. We need journalists and others to help us push back.

New comment by Firefishy in "OpenStreetMap overwhelmed by bots scraping data"

Firefishy — Wed, 28 Jan 2026 19:08:50 +0000

Disclosure: I am part of the OpenStreetMap mostly-volunteer sysadmin team fighting this.

The scrapers try hard to make themselves look like valid browsers, sending requests via residential IP addresses (400,000+ IPs at last count).

I reached out to journalists because despite strong technical measures, the abuse will not go away on its own.

New comment by Firefishy in "Slack has raised our charges by $195k per year"

Firefishy — Thu, 18 Sep 2025 13:34:34 +0000

OpenStreetMap Slack https://slack.openstreetmap.us/ was forced to downgraded to the free edition earlier this year for similar reasons.

New comment by Firefishy in "Future of OSU Open Source Lab in Jeopardy"

Firefishy — Thu, 01 May 2025 11:18:10 +0000

I help to run the OpenStreetMap infrastructure, and for many years the OSUOSL has generously hosted some of our servers.

Without OSUOSL OpenStreetMap would be more difficult to host and significantly slower to access from North America.

I hope OSUOSL can get the financial support they rightfully deserve.

New comment by Firefishy in "OpenWrt 24.10.0 – First Stable Release"

Firefishy — Thu, 06 Feb 2025 13:04:14 +0000

https://protectli.com/ Good quality devices. Real serial consoles to allow recovery when you make a networking configuration mistake ;-)

New comment by Firefishy in "Reverse-engineering and analysis of SanDisk High Endurance microSDXC card (2020)"

Firefishy — Sun, 02 Feb 2025 20:05:53 +0000

Raspberry Pi's "Making a More Resilient File System" document https://pip.raspberrypi.com/categories/685-app-notes-guides-... has instructions on how to configure the eMMC on the CM4 and CM5 to run in pSLC mode. Halving the storage capacity.

New comment by Firefishy in "Openstreetmap.org Is Offline for Days"

Firefishy — Tue, 17 Dec 2024 17:36:09 +0000

OpenStreetMap is now back up and running :-)

New comment by Firefishy in "OpenStreetMap's New Vector Tiles"

Firefishy — Tue, 19 Nov 2024 18:10:02 +0000

The new stack has some unique feature like the vector tiles being minutely updated directly from OSM mapping changes.

There are still issues to fix as it is still only a technical preview.

New comment by Firefishy in "OpenStreetMap's New Vector Tiles"

Firefishy — Tue, 19 Nov 2024 17:55:23 +0000

We don't offer 2x raster tiles because we simply don't have the resources to do it while keeping the tiles minutely updated and open access. We serve around 60k req/sec on a tiny donation backed budget.

The raster tiles are primarily for our OSM mappers to see their map changes rendered quickly to improve the feedback loop.

New comment by Firefishy in "Dear AI Companies, instead of scraping OpenStreetMap, how about a $10k donation?"

Firefishy — Tue, 30 Jul 2024 19:26:24 +0000

OP here. The toot was my sarcastic response after having to rate-limit and block another set of abusive scrapers aggressively hitting our website and mapping API. robots.txt be damned.

OpenStreetMap data is free to download. We published minutely on https://planet.openstreetmap.org/ and the data available via AWS S3 + torrent.

If you just starting out, best to start with a smaller regional extract: https://wiki.openstreetmap.org/wiki/Planet.osm

New comment by Firefishy in "Dear AI Companies, instead of scraping OpenStreetMap, how about a $10k donation?"

Firefishy — Tue, 30 Jul 2024 19:18:13 +0000

Our data is already publish via torrent, see https://planet.openstreetmap.org Our data, including minutely updates is available via public S3 buckets (EU & US) supported by the AWS Open Data Sponsorship Program.

New comment by Firefishy in "Dear AI Companies, instead of scraping OpenStreetMap, how about a $10k donation?"

Firefishy — Tue, 30 Jul 2024 19:14:34 +0000

We publish a live stream of minutely updated OpenStreetMap data in ready do digest form on https://planet.openstreetmap.org/ and S3. Scraping of our data still happens.

Our S3 bucket is thankfully supported by the AWS Open Data Sponsorship Program.

New comment by Firefishy in "A DRAM Failure"

Firefishy — Fri, 14 Oct 2022 08:56:16 +0000

FYI: https://github.com/mchehab/rasdaemon (replaced mcelog) is the daemon for watching for these ECC and other kernel reported "Reliability, Availability and Serviceability" errors.

rasdaemon also attempts to report which physical DIMM / slot triggered the ECC error.

New comment by Firefishy in "A Gov.uk site dedicated to porn?"

Firefishy — Thu, 25 Nov 2021 20:05:02 +0000

Sub-domain takeover attack. The sub-domain was CNAME'ed to a S3 bucket and the S3 bucket had likely been deleted. The porn purveyor, re-created a new S3 bucket with pr0n.

A scanner that would have caught the vulnerability: https://tech.ovoenergy.com/how-we-prevented-subdomain-takeov...

Or a grey hat scanner for finding sub-domains vulnerable to takeover: https://github.com/m4ll0k/takeover

New comment by Firefishy in "Ex-Netflix VP found guilty of awarding tech contracts in exchange for bribes"

Firefishy — Wed, 05 May 2021 10:46:18 +0000

Wikipedia updates pending ;-)

New comment by Firefishy in "Mapbox CEO says the map calling New York City ‘Jewtropolis’ has been 100% fixed"

Firefishy — Thu, 30 Aug 2018 19:21:29 +0000

The vandalism was reverted in 2 hours over 20 days ago. If Mapbox had stayed current with the diffs we publish, the vandalism would have been gone in a blink.

New comment by Firefishy in "Mapbox CEO says the map calling New York City ‘Jewtropolis’ has been 100% fixed"

Firefishy — Thu, 30 Aug 2018 19:18:14 +0000

Yes, here is the changelog: https://www.openstreetmap.org/changeset/61555047?node_page=3

New comment by Firefishy in "Mapbox CEO says the map calling New York City ‘Jewtropolis’ has been 100% fixed"

Firefishy — Thu, 30 Aug 2018 19:15:25 +0000

Correct. The vandalism was reverted within 2 hours and a data update publish a minute or 2 later. Mapbox import the original vandalism change only today for some reason and there seems to have been a delay in importing the revert.

New comment by Firefishy in "Quest-driven OpenStreetMap editing for Android"

Firefishy — Wed, 10 May 2017 13:11:41 +0000

Pretoria was mostly mapped by a single mapper very early in the project. There wasn't high quality aerial imagery available when he literally cycled every street in Pretoria.

Because the map "looks complete" Pretoria hasn't really attracted many mappers since his departure.

Cape Town has a much more vibrant community of mappers. I've also been able to get crazy high resolution aerial imagery for CT from the municipality which helps: http://coct.aerial.openstreetmap.org.za/

New comment by Firefishy in "AirPods are now available"

Firefishy — Tue, 13 Dec 2016 23:51:52 +0000

And the Plantronics BackBeat PRO headphones: http://www.plantronics.com/uk/product/backbeat-pro