Also another interesting point is their use of boosting, since i recently attended a tech talk by facebook engineers where they told us that bagged decision trees while predicting the friends for a user.

do they use physics based techniques, like black scholes?

or do they use statical learning based techniques?

Hadoop is more of an Distributed O.S. to run process and store data across multiple machines.

in fact Apple used 3.3.1 to make sure that code was not cross platform compatible.

While HN would seem extremely pro - entrepreneurship, in reality, starting your business is not always the smartest move, not from just personal perspective but from perspective of a nation as well. Sure you want Larry Page and Sergey Brin to create Google but before that they went to Stanford for Ph.D. in CS. Entrepreneurship just for sake of it and without any good idea is no better than playing stock market with different strategies.

     If 15% of the unemployed engineers in Corvallis could start companies that hired 5 full timers, they would (probably) have a booming economy all over again. Shit, maybe we could start exporting to China (I think they value entrepeneurship and industrialization there...).

The only way to generate income is via means which are non zero sum games [at least non zero sum within that nation].

Their supply is limited and controlled, anyone can come and say that they are "web programmer" after reading HTML/CSS/Ruby, Even Universities increase class sizes arbitrarily. All this is not possible in medical profession. Doctors are few, they are trained extensively, not anyone reading anatomy book can go for a residency!

1: He mentions infochimps, but according to my knowledge its more of an ebay for datasets rather than support/provider for Big Data Stack, also is it successful? I am unsure how infochimps is related to the Big Data stack.

2: From what I remember reading about 80legs, is that it uses distributed grid computing to run the crawlers (something like SETI @ Home), I doubt Hadoop was ever designed for such applications. So this is surely isn't a Hadoop use case.

3: Quoting:

       While the standard big data stack has made huge strides in making big data more accessible to everyone, it will always fall short against our stack when it comes to the cost of collecting data.  We actually don’t store that much data. Because 80legs users can filter their data on the nodes, they’re able to return the minimum amount of data in their result sets.  The processing (or reduction, pardon the pun) is done on the nodes.  Actual result sets are very small relative to the size of the input set.

Also he mentions """ We have about 50,000 computers using their excess bandwidth.""" 50,000?? The biggest Hadoop cluster That I know (Yahoo) has ~10-20k nodes, and Hadoop was never meant to be used at 50K scale for crawling. So they had no option other than building their own system, even if they had to build it today.

4: Quoting

      One advantage is optimization — an “off-the-shelf” system is going to have some generalities built into it that can’t be optimized to fit your needs. The opportunity cost of going “standard” is a slew of competitive advantages.

While I seem to agree with Authors conclusion that a highly focused startup should make their proprietary solution, I cannot agree with his evidence behind that argument. A grid based crawler with 50K machines isn't something that Hadoop was ever designed to support.

Well you can start with reading this book http://www.cs.cornell.edu/home/kleinber/networks-book/ for overview, regarding application of these techniques are considered you can look for papers at recent WWW, NIPS,ICML conferences. The most popular and well studied areas are Link Prediction and Community Detection. The SNAP library comes with some good example code. You can also have a look at Divisi project at MIT Media Lab if you are interested in reasoning/ analogy over the networks.

For real datasets, there are a lot of encyclopedic datasets such as Wikipedia / DBPedia /Semantic Web/ Music Brainz, as well as social ones such as Twitter follower network dataset. If you are in a university, you can even get full Web Graph from Yahoo [for research use alone].