Like I said in the original comment: this isn't (to my knowledge at least) pure mathematics that's being kept secret. But there are absolutely families of techniques and algorithms whose applications to finance are nontrivial, non-incremental and very well guarded.

Who are "these guys"? The funds discussed in the article have average annual returns well above 9%.

I used to be part of a research group that sold the so-called "alternative data" you're describing to 30 or so hedge funds in the NYC area, including several of the largest. The example I like to give is that we knew well ahead of time that Tesla would miss on the Model 3 because we knew every vehicle they were selling by model, year, configuration, date and price with <99% accuracy. I still occasionally sell forecasts like this and the methodology is straightforward enough that even a solo investor can consistently beat the market if they know how to source the data. But I've mostly lost faith in this technique as the sole differentiator of a fund's alpha.

Some funds, like Two Sigma, have large divisions with a very sophisticated pipeline for this kind of analysis. They do exactly what you describe. For the most part it works, but there are several obstacles that keep this from being the holy grail of successful trading:

1. First and foremost, this analysis is fundamentally incomplete. You are not forecasting market movements, you're forecasting singular features of market movements. What I mean by that is that you aren't predicting the future state of a price; if the price of a security is a vector representing many dimensions of inputs, you're predicting one dimension. As a simple example, if I know precisely how many vehicles Tesla has sold, I don't know how the market will react to this information, which means I have some nontrivial amount of error to account for.

2. This analysis doesn't generalize well. If I have a bunch of information about the number of cars in Walmart parking lots, the number of vehicles sold by Tesla (with configurations), the number of online orders sold by Chipotle, etc. how should I design a data ingestion and processing pipeline to deal with all of this in a unified way? In other words, my analysis is dependent upon the kind of data I'm looking at, and I'll be doing a lot of different munging to get what I need. Each new hypothesis will require a lot of manual effort. This is fundamentally antagonistic to classification, automation and risk management.

3. It's slow. Under this paradigm you're coming up with hypotheses and seeking out unique and exclusive data to test those hypotheses. That means you're missing a lot of unknown unknowns and increasing the likelihood of finding things that other funds will also be able to find pretty easily. You are only likely to develop strategies which can have somewhat straightforward and intuitive explanations for their relationship with the data.

This is not to say the system doesn't work - it very clearly works. But it's also easy to hit relatively low capacity constraints, and it's imperfect for the reasons I've outlined. You might think exclusive data gives you an edge, but for the most part it does not (except for relatively short horizons). It's actually extremely difficult to have data which no other market participant has, and information diffusion happens very quickly. Ironically, in one of the very few times my colleagues and I had truly exclusive data (Tesla), the market did not react in a way that could be predicted by our analysis.

The most successful quantitative hedge funds focus on the math, because most data has a relatively short half-life for secrecy. They don't rely on the exclusivity of the data, they rely on superior methods for efficiently classifying and processing truly staggering amounts of it. They hire people who are extraordinarily talented at the fundamentals of mathematics and computer science because they mostly don't need or want people to come up with unique hypotheses for new trading strategies. They look to hire people who can scale up their research infrastructure even more, so that hypothesis testing and generation is automated almost entirely.

This is why I've said before that the easiest way to be hired by RenTech, DE Shaw, etc. is to be on the verge of re-discovering and publishing one of their trade secrets. People like Simons never really cared about how unique or informative any particular dataset is. They cared about how many diverse sets of data they could get and how efficiently they could find useful correlations between them. The more seemingly disconnected and inexplicable, the better.

Now with all of that said, I would still wholeheartedly recommend this paradigm for anyone with technical ability who wants to beat the market on $10 million or less (as a solo investor). A single creative and competent software engineer can reproduce much of this strategy for equities with only one or two revenue streams. You can pour into earnings positions for which your forecast predicts an outcome significantly at odds with the analyst consensus. You can also use your data to forecast volatility on a per-equity basis and sell options on those which do not indicate much volatility in the near term. Both of these are competitive for holding times ranging from days to months and, with the exception of some very real risk management complexity, do not require a large investment in research infrastructure.

2. It's not just my opinion, and

3. I didn't say they're "well ahead" unilaterally.

This isn't unique to finance; industry labs in tech also often have novel results in applied mathematics and computer science that are ahead of academia and other industry labs. You don't have to believe me but it's not exactly a controversial topic. Not everything is published or patented.

> “SANDHOGS,” THEY CALLED THE LABORERS who built the tunnels leading into New York’s Penn Station at the beginning of the last century. Work distorted their humanity, sometimes literally. Resurfacing at the end of each day from their burrows beneath the Hudson and East Rivers, caked in the mud of battle against glacial rock and riprap, many sandhogs succumbed to the bends. Passengers arriving at the modern Penn Station—the luminous Beaux-Arts hangar of old long since razed, its passenger halls squashed underground—might sympathize. Vincent Scully once compared the experience to scuttling into the city like a rat. Zoomorphized, we are joined to the earlier generations.

This goes on for about seven paragraphs before I have any idea what the article about. I understand “setting the scene” but I can’t tell whether or not to care about an article if it meanders about with this flowing exposition before indicating what its central thesis is.

It seems like a popular style in thinkpieces and some areas of journalism. The author makes a semi-relevant title, provacative subtitle, and five - ten paragraphs of “introduction” that throw you right into the thick of a story whose purpose doesn’t seem clear unless you know what the article is about. Rather than capturing my attention with engaging exposition, I find it takes me out of it. But it must work if it’s so uniquitous; presumably their analytics have confirmed this style is engaging.

This is not a correct representation of liquidity, and thinking about it under this definition can be very dangerous. You need to consider:

1. How many shares are there outstanding?

2. What is the ask price of those shares on paper?

3. What is the bid price of those shares by investors willing to purchase them on the private market?

4. How many owners are allowed to sell their shares at the same time?

5. How many owners could realistically find a buyer at the paper ask price of the shares?

6. How many owners could sell their shares before the existing deviation (spread) between the paper ask price and available bid prices changed?

This is not to say your overall point is wrong, it's to say that it can't be defended this way; more importantly, we really shouldn't be simplifying our discussion and its definition of liquidity to the one you've presented here, which is too simplistic. There is a lot of nuance about price discovery between public and private valuation that's missing here. For (one) example, you can maintain an artificially inflated valuation of a private company if there are fewer owners willing/able to sell than there are buyers, despite a relatively larger set of potential owners either not allowed to, or not conveniently capable of, selling their shares. This scenario makes presents an asymmetry between the weighting and availability of positive vs negative price sentiment that is much more easily resolved in the public market.

Otherwise I think point about the $80B not being liquid is a good one. It's not a dishonest figure, but it's clearly inaccurate and inappropriate for the purpose of estimating returns. The real answer is going to be far more nuanced than simply stating the aggregate value of all YC companies on paper.

That these figures are not public is a very important discussion point, because it does introduce some level of anecdata and arbitrary speculation into the discussion. That's not to say you're wrong, but it's certainly imprecise and questionable.

Even if the process is inefficient (many false negatives), if it yields a population with fewer false positives than the general population it is meaningful. This is an important distinction because many people seem to be unilaterally declaring these hiring practices to be useful due to dislike and anecdata. Of course they're not ideal and there is a lot of room for improvement, but there's no empirical reason for us to act as though the entire system is arbitrary just because we don't like it.

The primary question we should be looking at to inject some empiricism into this discussion is how many engineers end up being fired once hired, how long it takes to fill each technical role on average and how many candidates are turned down. There seens to be a false dichotomy at play, where people are only able to damn the process in its entirety, but it can still be bad and retain some signal. That shouldn't be a controversial point, it should be a minor preliminary observation.

This implies that the capability distribution at Google is approximately equal to the capability distribution of the entire set of eligible developers. I don't personally think that's true, and in any case even if it is true it's not obvious.

I also don't think it's fair to characterize most companies as believing their recruiting practices infallible. Google's former director of recruiting has frequently talked about how their recruiting processes (and particularly interviewer feedback) seemed very noisy and fairly inconsistent in a variety of dimensions. The common recruiting-page-positive-vibes spiel might indicate a lot of confidence, but that's not what I would use to determine it. A lot of companies also seem pretty happy to try out new programs like e.g. Triplebyte or otherwise innovate on their sourcing processes.

I don't think your point about precisely quantifying the definition of a "great developer" is a good one. I can capture an observation of superior capability in several dimensions without having to rigorously quantify the relative differences in capability or precisely why one is more capable than the other. If one developer accomplishes in a few days what takes another two weeks with code that is at least as maintainable and performant, that developer is better. If that developer is better when put next to most of the other developers you have around you, then they're a "great developer."

Your focus on quantifiable definition seems to imply that significant differences in human capital can only be ascertained mathematically, but that doesn't reflect quite a lot that we're already familiar with in everyday life. If I'm placed in front of two walls which are both much taller than I am, I can see which of the two is taller than the other if the difference is significant as long as the tops aren't out of sight. I don't need to quantify this; I could just say, "One looks to be a lot taller than the other, but I'm not sure by how much exactly, or what their respective heights are."

I'll happily agree that a lot of tech interviewing (and interviewing in general) is dreadfully broken. I'll also happily agree that figuring out which skills to prioritize and how to judge the level of those skills in candidates is very difficult work. But I strongly disagree that the fundamental premise of that work is intrinsically "voodoo." If your bar is a mathematical formalization then of course you're going to be disappointed, but that also holds true for the majority of our professional and personal activity.