In the early 20th century, US rail companies were beholding a very favorable situation: high demand to run loads of heavy freight all over the country, high demand to ferry passengers all over the country, and basically no serious competitors to either revenue source.

Now freight revenue was never going to be transformative to the industry, but it had the benefits of being reliable, un-fussy, and fairly easy to build a financial business around. Passengers, on the other hand, offered huge revenue potential, but had the downsides of being very fussy about things like safety and comfort and timeliness, along with wanting stations in convenient places and an ever-expanding rail network.

Students of US business management history should be unsurprised, then, that while evaluating the market that offered reliable revenue, versus the market that wanted large capital investments, the railroads overwhelmingly chose the freight market. In other words, US the railroad companies spoke and said we do not want passengers loudly and clearly.

The thinking was: passengers can do take the wagons and busses and cars and these newfangled airplane thingies, but freight is a guaranteed market for us! So the passengers slowly migrated to other form of transportation. But the kicker was, freight also wanted things like timeliness and access to an expanding transport network and, shockingly for the railroad execs, were willing to pay for it.

Add about 80 years, declining rail traffic, and tons of corporate mergers, and we have the sad state of US railways today: many residents have never seen a railway expansion or shiny new rail equipment, much less a real functioning passenger train. It's easy and comfortable to say that zoning or regulations or market forces allowed US rail to languish, but that would be ignoring the part where the industry did not want the customers in the first place.

To the second point, the race to the bottom won't be evenly distributed across all markets or market segments. A lot of AI-economy predictions focus on the idea that nothing else will change or be affected by second and third order dynamics, which is never the case with large disruptions. When something that was rare becomes common, something else that was common becomes rare.

Generously, this writing style is supposed to show the business value of teams and individuals, for promotions or other recognition. But yeah, it can be frustrating to read this style.

Look, when we break the feedback loop back to the people who wrote the software in the first place, they get happier for a bit, you make some other people sadder for a bit, and then slowly your feature crew never want to be interrupted or bothered again and your customer crew can't get enough resources to fully fix anything.

Worse, your feature crews aren't learning anything beyond how to get those lines out the door, which will somehow get slower and more expensive as time goes on. Why? Because you removed the one fitness function on good software development which is to fully re-incorporate the negative feedback back into the source of development.

A real CTO leadership handbook would say clearly "it's your responsibility to help your developers improve, especially while shipping, and they're not always going to be happy about it."

The easiest way to explain this is with a simple sequence of events: the database has a temporary issue, system capacity drops, clients start timing out or getting errors, load amplification kicks in with retries and request queueing, load is now higher than normal while capacity is lower than normal, devs work hard to get the database back in order, database looks restored but now the system has 3x the load it did before the incident, other heroic efforts are needed to shed load and/or upscale capacity, whew it's all working again! In the post-mortem there are lots of questions about why rate-limiting didn't protect the system. Unfortunately, the rate limit values required to restore the saturated system are far too low for normal usage, and the values needed for normal operation are too high to prevent the system from getting saturated.

Fundamentally, there's really no way for a rate limiting (which only understands incoming load) to balance the equation load <= capacity. For that, we need a back-pressure mechanism like circuit breaking, concurrency limiting, or adaptive request queueing. Fortunately, rate limiting and back-pressure work well together and don't have to know about each other.

1. Rate limits don't really protect against backend capacity issues, especially if they are statically configured. Consider rate limits to be "policy" limits, meaning the policy of usage will be enforced, rather than protection against overuse of limited backend resources.

2. If the goal is to protect against bad traffic, consider additional steps besides simple rate limits. It may make sense to perform some sort of traffic prioritization based on authentication status, user/session priority, customer priority, etc. This comes in handy if you have a bad actor!

3. Be prepared for what to communicate or what action(s) to perform if and when the rate limits are hit, particularly from valuable customers or internal teams. Rate limits that will be lifted when someone complains might as well be advisory-only and not actually return a 429.

4. If you need to protect against concertina effects (all fixed windows, or many sliding windows expiring at the same time), add a deterministic offset to each user/session window so that no large group of rate limits can expire at the same time.

Hope that helps someone!

If the data is in fact updated with clear hot/warm/cold sets, caching the cold sets should be extremely effective, the warm set moderately effective, and it may not even be worth caching the hot set at all, given the complexity proposed. Additionally, you should be able to offload the cold sets to persistent blob storage, away from your main database, and bulk load them as needed.

Finally, it can be faster and simpler to keep track of deltas to cold sets (late mutations that happen to "invalidate" the previously immutable data), by simply storing those updates in a separate table, loading the cold set data, and applying the delta corrections in code as an overlay when queried. Cron jobs can read those deltas, and fold them back into the cold set aggregations, making clean validated cold set data again.

Great article, BTW! There are entire database technologies and product dedicated to addressing these use cases, particularly as the data sets grow very large.

For actual scheduled events, we just share our personal Google calendars with each other and invite each other to mutual events. No issues there.

The largest technical issue currently is just organizing personal media in a reasonable way. We're in the in Apple ecosystem so shared albums have generally worked for photos and videos, but it's clunky at best and doesn't cover other media or documents.

  What I wish I knew was that if you’re an early company employee, it’s not
  likely that the skills you have on day one are the skills needed as the
  company scales to the next level. This sentence is worth reading multiple
  times as no one – not the person who hired you, the VC’s or your peers -is
  going to tell you when you’re hired that the company will likely outgrow you.

No, it's not natural for a grown-ass adult to lose it. Many people can face loss, disappointment, or criticism without losing it. Elon doesn't need to lose it and neither Tesla, Space X, The Boring Company, nor the Thai cave rescue effort were aided in any way by Elon's self-centered histrionics.

We don't need to pretend that his drive to be a visionary entrepreneur also requires him to lash out when he feels personally wounded, nor should we support attempts to rationalize this behavior.

Imagine how much more effective Elon would be in his professional life if he had a bit more self-control.

The largest source of your tech debt may in fact be that brilliant one writing volumes of code destined to become the collective headache of future dev teams.