I guess the real question here is how are they able to steal from you? Were they purchasing gift cards from a merchant with lax security?

It’s one thing to guess a number it’s another thing to get the money out of the system

At extremely high scale you start to run into very strange problems. We used to say that all of your "Unix Friends" fail at scale and act differently.

I once had 3000 machines running NTP sync'd cronjobs on the exact same second pounding the upstream server and causing outages (Whoops, add random offsets to cron!)

This sort of "dogpile effect" exists when fetching keys as well. A key drops out of cache and 30 machines (or worker threads) trying to load the same key at the same time, because the cache is empty.

One of the solutions around this problem was Facebook's Dataloader (https://github.com/graphql/dataloader), which tries to intercept the request pipeline, batch the requests together and coalesce many requests into one.

Essentially DataLoader will coalesce all individual loads which occur within a single frame of execution (a single tick of the event loop) and then call your batch function with all requested keys.

It helps by reducing requests and offering something resembling backpressure by moving the request into one code path.

I would expect that you'd have the same sort of problem at scale with this system given the number of requests on many procs across many machines.

We had a lot of small tricks like this (they add up!), in some cases we'd insert a message queue inbetween the requestor and the service so that we could increase latency / reduce request rate while systems were degraded. Those "knobs" were generally implemented by "Decider" code which read keys from memcache to figure out what to do.

By "pushes to connected SDKs": I assume you're holding a thread with this connection; How do you reconcile this when you're running something like node with PM2 where you've got 30-60 processes on a single host? They won't be sharing memory, so that's a lot of updates.

It seems better to have these updates pushed to one local process that other processes can read from via socket or shared memory.

I'd also consider the many failure modes of services. Sometimes services go catatonic upon connect and don't respond, sometimes they time out, sometimes they throw exceptions, etc...

There's a lot to think about here but as I said what you've got is a great start.

During some thread, some where, there's going to be a roundtrip time between my servers and yours, and once I am at a scale where this sort of thing matters, I'm going to want this on-prem.

What's the difference between this and checking against a local cache before firing the request and marking the service down in said local cache so my other systems can see it?

I'm also concerned about a false positive or a single system throwing an error. If it's a false positive, then the protected asset fails on all of my systems, which doesn't seem great. I'll take some requests working vs none when money is in play.

You also state that "The SDK keeps a local cache of breaker state" -- If I've got 50 servers, where is that local cache living? If it's per process, that's not great, and if it's in a local cache like redis or memcache, I'm better off using my own network for "sub microsecond response" vs the time to go over the wire to talk to your service.

I've fought huge cascading issues in production at very large social media companies. It takes a bit more than breakers to solve these problems. Backpressure is a critical component of this, and often turning things off completey isn't the best approach.

Buy a coffee, get a new password, etc.

there is comcast for high speed, and well… there is comcast.

If you want slow bonded DSL you can go to Sonic. Or At&T.

Only recently has LTE become fast enough to make home Internet over LTE an acceptable alternative

in the 1990s I helped build a dating site where people put their profiles on small voicemails. This is very reminiscent of that and I think the engagement was low because it was a lot to listen to

any chance you’ll open source this?

My boards and code live here:

https://github.com/netik/rpi_gameshow

The article starts by blaming then iPhone and social media and goes on to show how the child is a victim of poor parenting and divorce.

Maybe the child’s depression, anxiety, and longing for acceptance comes straight out of the broken home and not social media.