What happened was that at some time around the end of 2019 the credit card expired and nobody noticed it. The API just kept on working for our services, so we assumed everything was fine and we were probably within the free tier. The card was added and account configured by someone not working on the project anymore.

Then recently the service just stopped working and after a bit of digging around, we found that the account had racked up a bill of $20.000 since the end of 2019. Slowly and steady each month. It was nice enough of Google to give us quite the credit period there, but as it turns out after inspecting the cost breakdown, someone else must have been using the key since a lot of the requests were to API endpoints totally unrelated to the services we provide.

We screwed up big time by having an unrestricted key configured. It was setup in 2017 way back when things were free, so I doubt anybody actually thought about abuse of the key at that time, but none the less, here we are having our key abused.

Do we even have a chance here? Did anyone get into a similar situation? I would be grateful for a little bit of advice about how to approach Google. I realize this is our fault, but at the same time, it seems we never got any payment reminders or notifications that we had bills overdue - at least we cannot find them anywhere.

Comments URL: https://news.ycombinator.com/item?id=26439591

Points: 18

# Comments: 25

Is what they enable worth $40.000 to the enthusiasts?

I got out because it turned into exactly a religion. There was no room for questioning the technology or considerations about whether this was really as useful as we thought it would be.

Went to an independent shop and had the problem fixed for $100.

Fuck Apple.

I worked at Apple and it was absolutely horrible. I remember sitting on the train and actually thinking about killing myself. Reasons were many, but mostly I just didn't fit in there, I think.

I felt like people there were also a bunch of whiny little bitches to be honest. Always having some problem with a hand or a back or some family problem or whatever which demanded that they take days off. There were a few people whom I felt were probably talking some shit behind my back, but I couldn't really believe it until I read this story. It was just a horrible horrible place and I can also recognize that thing about having a predecessor, who was apparently a "bad person" and thus had to be terminated.

Every other day I would have people telling me how great a place it was and how lucky we were to be working in a place so big while still having that startup feeling. I have worked at startups and I always just imagined to myself that 40 year old "dude" who refuses to grow up. That is Apple..

Never again.

I got dragged through the whole white boarding experience also for the first time and it was fun enough, but in hindsight, they should just have asked me if I would suck Steve Jobs dick, if he came in that door. The answer to that question alone would have provided them with enough information about whether I would have been a good hire or not.

I am 45 and have never every experienced a work place like that. Always had positive feedback and never had any problems with colleagues.

Its pretty much the worst thing that can happen to systems that are not supported by laws of nature. Fictional structures implode once trust is gone. Financial systems, ideologies, religions.. Gone..