For instance, a web crawler. Establish your boundaries right away. You can't crawl the entire web right out the gate, so you need to decide where to stop. 1M pages? 1B? Does your crawler run continuously, or in batches, on a schedule? Are you concerned with avoiding crawling duplicate links? How often do you want to revisit links you've seen before? Are you going to limit page sizes you'll keep? Are you keeping images and other files, or only HTML content? Do you need to be able to execute JavaScript? How do you want to access the results: search, machine learning, analytics, etc? How many requests for those results do you expect in a given time frame? What's the maximum time you'd like to wait for a response?

All of these questions will affect your system design.

Take those answers and do some napkin math to figure out the scale you're working with.

If you are crawling only 1M pages and keeping only the HTML of pages 50kB in size you can easily use any database off the shelf with little tuning, for pretty much any use case (modulo response time requirements). For allowing larger files and images you'll probably want to reach for commercial object storage (unless you're a whizzkid file server guy), and if you're crawling a billion links you'll likely need to think carefully about horizontal scale and partitioning, and if you need to provide search or analytical processing you'll likely want to bring in specialized data stores, which means setting up some pipes to connect them through replication or batch processing...

It goes on and on like that. You need a broad knowledge of the capabilities and limitations of different softwares across various categories to understand what your options are and when the complication of adding them would be warranted. I find it helps to understand the historical context under which they were first developed, so that you can understand the problem they were trying to solve.

In general, I've had the misfortune of not being the dumbest guy in the room at any given time, so I've not received much in the way of mentorship. I've had to mostly learn by doing, which gives me a particular bias towards building over buying and avoiding bringing in new software dependencies where I can help it.

I encourage you to learn by doing as well, even if you stop around the first or second roadblock, or about the time you start needing to spend your own money to go further. Maybe try implementing things locally in a way that doesn't let you do more from your personal machine per se, but instructs you in the complication of setting up e.g. kafka or elastic search, and their benefits and tradeoffs.

Think hard about how you would monitor the system to make sure it's working well, think about the dashboard you'd want to see to assure you of the health of the system. Think about where the bottlenecks to performance would be. Think about what you'd change to ensure it stays up 99.999% of the time, even when you're deploying changes to it or scaling up the database. Think about what you'd need to do to ensure that you never lose data.

And then read other people's experiences, but remember to take a grain of salt with you. People like to talk a big game, you'll hear people say that tool XYZ is the ONLY choice for a serious project and... that may or may not be true. Depends on the scale.

But usually in system design interviews people want you to whip out all the keywords and show you know why you're using them and what they're for. So when prepping for those, don't be afraid to say "streaming CDC events with debezium and kafka" instead of "batch job loading new records from postgres" or w/e, as long as it's applicable and you can justify it.

System design interviews also tend to revolve around a few categories of problems (web crawling/search, Google docs, building social media feeds, etc etc) so reading other people's answers for those will start giving you the flavor.

I give the system design interviews at my company (well, now I mostly train other people how to do them) so usually I look for people to ask this constraint questions, show breadth of knowledge, reveal where they're strongest by going deep on a few topics, and the more senior they are the more I want them to lead the conversation and ask me questions rather than the other way around. I don't really expect people to come in and deliver a rote-memorized google-scale answer off the top of their head, and tbh I'd be really suspicious if they did.

The "chance at making FU money" and "100% more work/stress" and "losing funding or... goes bust and can't make payroll", that's stuff that happens at startups that are paying _below_ the VC-backed market rate.

Once a company gets money to be paying employees in the third hump of the trimodal distribution of SWE pay, those "exciting" parts are long gone. You're no longer in the running to receive a hilarious amount of money should everything 10-100x, and you're no longer worried about going bust and not making payroll.

Even a pre-series A startup I worked at saw the writing on the wall with 5 months of runway left and switched from trying to get funded to finding an M&A deal. At the very least you can get the team acquired by someone in the expansion phase and get the rare chance to interview as a group and compare notes and have an exec try and negotiate your salary _up_ for once.

By the time a startup is paying VC-backed market rate, the level of stress you have just comes from the usual suspects: management, culture toxicity, etc. some of that you can absorb, some might prompt you to find a new job, just like normal.

There's still a level of "hoo-rah, we're a startup" present of course, but I just consider this a bad attempt to keep the magic of when the company and products were small alive. I let it affect me none.

We were pretty large, big-name online merchandise retailer. Definitely put our heels to the fire on giving the frontend a design refresh.

The law references that you should strive to follow all standards in your own output, but you should make a best effort to accept content that may break a standard.

This is useful in the context of open standards and evolving ecosystems since it allows peers speaking different versions of a protocol to continue to communicate.

The assertion being made here is that the world has become too fraught with exploiting this attitude for it to continue being a useful rule

Tracking the latest release is important because:

1. Distributions build (most? all?) libraries from source, using compilers and flags the algorithm authors can't control

2. Today's latest release is the base of tomorrow's LTS.

If the people who know most about these algorithms aren't tracking the latest compiler releases, then who else would be qualified to detect these issues before a compiler version bearing a problematic optimization is used for the next release of Debian or RHEL?

> Logically, therefore, must we not also expect CPU designers to also forego changes that could alter timing behavior?

Maybe? [1]

> freezing all compiler development

There are many, many interesting areas of compiler development beyond incremental application of increasingly niche optimizations.

For instance, greater ability to demarcate code that is intended to be constant time. Or test suites that can detect when optimizations pose a threat to certain algorithms or implementations. Or optimizing the performance of the compiler itself.

Overall I agree with you somewhat. All engineers must constantly rail against entropy, and we are doomed to fail. But DJB is probably correct that a well-reasoned rant aimed at the community that both most desires and most produces the problematic optimizations has a better chance at changing the tide of opinion and shifting the rate at which all must diminish than yelling at chipmakers or the laws of thermodynamics.

[1]https://en.m.wikipedia.org/wiki/Spectre_(security_vulnerabil...

The LLVM backend compiles this IR to the target architecture's machine code.

Some optimizations can (and do?) happen in the backend compiler, but for the most part it is up to the frontend to emit IR that is well optimized within the semantics of the frontend language.

The history of Rust, for example, is filled with examples of optimizations made not to the standard library, but to how the frontend emits IR.

Beyond that, different languages and ecosystems are going to have differing levels of optimization in their stdlib and runtime that will have an outsize effect on day-to-day performance.

My understanding is that the FBI or other non-judicial body cannot unilaterally issue a gag order. Subpoenas and gag orders related to them are granted by judges.

(Which isn't to say that the relationship between the judicial branch and law enforcement bodies is always pure and equal)

For that application at least no changes to the runtime would be necessary

"Open Core", where the core product is open source and you provide closed source extensions, generally high-value integrations and enteprise-y features like SSO, active directory support, etc.

"Paid Services" where you have offerings on top of the open product. You might offer trainings, consultations, support contracts, custom development work (e.g. aforementioned high-value integrations). Also very common is to offer a fully-managed SaaS experience, but I wouldn't expect that to be your key to success; many open-core businesses fail expecting that full hosting to be more attractive than it is.

It's becoming more popular to offer "on-prem"/"self-hosted" support. Cloud costs are going to get cut after 3rd party products among your customers, so getting your product counted in column B gives you a little edge. Even if they need to cut their support contract down they're still using your product and are more likely to come back when the storm is over.

Best practice and good engineering in web dev is the same as it is in any other software field: the best stack is the one you know cold.

Unfortunately, there are many popular stacks, and all of them have something to say about all the others. And 80% of learning materials for any given stack aren't from official sources, but are instead blog posts and videos and tweets from tech influencers. When they parrot those same criticisms, it lends the air of a grassroots shifting of the whole community.

The fact of the matter is that despite the huge levels of hype flux, large, popular, stable apps are still being made with technology "boring" tech.

Despite SPAs, rails and Django apps are still built rendering server side templates and sprinkling jQuery where necessary. Despite GraphQL, REST APIs are still the state of the art for most APIs. Despite kubernetes and docker, many apps are still deployed to heroku or VPS providers.

It can be intimidating for newcomers because it can be difficult to separate hype from fact. My advice is to pick something, the more boring the better, and know it cold and ignore all claims of its death.

To address concerns about projects dying, that is pretty legitimate. Unfortunately webdevs deploy trusted code into a highly untrustworthy and antagonistic environment. New technologies do come to play and requires adapters and plugins to be written. I wouldn't worry about overall size of community as long as security issues are getting addressed.

It was designed to power experiences that look like shopping carts: relatively short-lived, mostly append-only, needing a flattish performance curve as the number of items and users go up, with a really natural partition key.

Using it as the primary storage of your app means you play the part of query planner a lot of the time.

If your data is especially graph-structured you might end up doing so-called "single table design" where all objects end up in the same table that has its keys constructed carefully (compound keys formed by string concatting), And has secondary sparse indices to power other access patterns.

And definitely plan on overfetching as an _optimization_ to prevent many small queries to dynamo, especially in a GraphQL context where we have no idea what fields are being requested at fetch time.

But your point about on-prem stands

Pretty much any book by Greg Egan covers these ideas since nearly all of his stories take place somewhere in a shared universe at different points of humanity approaching the singularity. Check out his short stories in Axiomatic or the full length novels Permutation City and Schild's Ladder.

However, to advocate for blogging I've got 2 points:

1. Refining your opinions -- related to becoming a better communicator, I think that forming a cogent opinion post requires you to really think through your thoughts, opinion, and experience on a topic. As a project it can be a great learning exercise, I find that I gain a much more solid understanding once I feel the need to back up my opinions with experience and references.

Which brings me to 2. Learning in public.

Taking the time to write up a good post and share it around exposes you to people who will agree with you, challenge your ideas, and importantly learn from your experience. It's a way to find a community and give back to it, and gain more in return.

Does 1 help you with c? Yes. Is 2 also networking, branding, maybe building a following? Sure, if you try at those parts.

Most the blogs that I want to read care a lot more about concentrating knowledge and experience so they can learn from and help grow others e.g. Julia Evans or Alexis King

The author was being sincere, rather than sarcastic as your initial parse suggests