In short, 5-10 years ago, writing mapreduce / spark jobs (or even debugging / optimizing hive jobs) was complex enough that it was often the job of the data engineer (and not the data analyst / scientist). And I do not only mean writing the data processing logic, but more importantly, properly configuring it so that the resource footprint was acceptable. This required a good understanding of the underlying framework, analyzing the job execution plan, tweaking the resource configuration, etc.

Now, writing distributed jobs is pretty trivial with most cloud providers, hence it is now purely done by data analysts and scientists. And the data engineers have switched to doing more of a devops kind of work, doing the plumbing between the various cloud components and the IaC required to provide those cloud resources to other data users. In short, you can be a data engineer and have absolutely no clue on how distributed systems are actually working, this will not be an issue in your daily job.

And even as an engineering manager, I do not feel safe. I think only once you reach director level, you are protected from market hype and newest frameworks trends.

Which is btw one of the depressing thing for a lot of data engineers: we used to play with those cool distributed processing frameworks, and now? We are mostly writing some terraform to deploy cloud resources, most of the distributed part being handled by those cloud providers.

Or is it just that in tech, we have the luxury to think about all of that, and to potentially take several years without working much thanks to our high salary?

Looking around me, I do know a fair amount of people quitting tech. To become a sport or management coach, a sound designer, a bar owner, a writer, etc. But a lot of those people seem to struggle even more now than they were when being in tech.

Never heard that there were crazy interview rounds for experienced lawyers, marketers, project managers, even other non-tech engineers. After a certain amount of experience, people assume from your resume that you are not an impostor and interviews are mostly about motivations / behavourial fit.