The bitter lesson is that simply scaling training on more games—including self-play—trumps any hand-crafted human input, whether that's fine-tuning on human commentary or clever engineering tricks.

Current models are just high-dimensional interpolation engines. The denser the data sampling, the more accurate the interpolation gets. Given a choice between denser sampling and anything else, denser sampling always wins. That is the bitter lesson.

Computer chess is the canonical example of this.

Comments URL: https://news.ycombinator.com/item?id=46865509

Points: 3

# Comments: 0

Sorting-networks which were already mentioned seems similar but a bit too abstract.

My code above doesn't contains values but those are easy to add I think. Of course it is better to permute fixed size pointers / offsets and not the entire blobs which can be of variable size and then it will complicate everything beyond feasible for SIMD

But I'm not sure, it's my interpretation. The write-up resonates with me but I can see it as a bit ambiguous and inviting many incompatible interpretations.

Proper treatment of the subject in a book format would be great.

However, Intel dominance is not to be underestimated, they definitely can make industry wide impact quickly. Just saying you can not easily draw parallels here

Why entire shared hosting industry custom-patches their kernel constantly with OpenVZ and stuff?

Why shared hosting has such a bad reputation for being insecure (despite all efforts).

1. Small surface for attack. With NaCl it is all concentrated in single tiny validator module. The model is also simple and mathematically proven to be secure. 2. Prior testing. It is especially hard for security product. Establishing motivating prizes is good way to ensure it is not easily breakable. 3. The speed with which patch is made available 4. Defense-in-depth, ability to have multiple levels of defense cheaply.

2) All these is impossible in ZeroVM except accessing memory randomly and thrashing caches and TLB tables. Hm... that could work, I guess. For the first time in this forum we talk about real vulnerability. However, I think the problem exists also in KVM/XEN (will do a proper research now, Googling EC2 TLB thrashing doesn't yield anything interesting), no access to other tenant data just temporarily slowing down specific processor chip.

If you meant more practical uses for it then unfortunately modern JIT would be difficult to support efficiently as they constantly recompile and with ZeroVM it is not only recompilation but also validation. However, JIT that recompile only once, on loading, is easy to support. In fact, next version of NaCl dumps GNU toolchain in favor for JITy LLVM, but then recompilation is happening only once.

1. Hadoop runs on EC2 2. Data is stored on S3 3. Intermediary results stored in EC2 4. Hadoop loads the data from S3 to EC2 5. EC2<->S3 bandwidth is not that fast or efficient (S3 proxy, network contention, TCP/IP processing)

Hypothetical MapReduce/ZeroVM/Swift scenario:

1. Data is stored on S3/Swift 2. Map and Reduce functions are run inside S3/Swift secured by ZeroVM in majority of cases accessing data locally without networking/proxies getting in the way. 3. Intermediate and final results are also stored within S3/Swift. 4. Local data access is efficient, fast and predictable 5. Local networking within S3/Swift is more efficient, fast and predictable than S3<->EC2 / Swift<->Nova

Accelerated Hadoop scenario:

Exactly as in #1, just Hadoop makes "predicate pushdown optimization" into S3/Swift secured by ZeroVM.

Regarding 'due to security restrictions' I meant that cloud vendor would not let you run your own code in S3 or CloudFiles. Why? Because you could mess up other people data and storage system itself. Why not run in VM inside S3? well I guess it would be impractical due to long provisioning time of conventional VM.

Kernel is such vast area vulnerable for an attack that it is scary even to think about securing all of it and not leaving a single weak point. Moreover, you will screw your syscall API to the point that it will become unusable. At bare least we need standard for the syscall capping and etc... so programmer will know what to expect.

And thanks for the link, will check them and what solution they use and whether they are happy with it.