When I was 15 or so, mid 2000s, I was heavily into the Blender community. Shortly after the closed source to open source transition I ran into a segfault. I could half heartedly sling code at the time but not enough to work out the issue I had.

After noting the issue on IRC Ton personally helped tease out the bug. I _might_ have misdirected him by saying the bug occurred "with just a cube" (and he was like "Yay! Bugs are easier when they're simple cases!") but I neglected to note it was a cube subdivided half a dozen times, with a few thousand vertices ;)

He had a vision and quiet persistent execution plan for Blender and the community that was far ahead of anything else in those (relatively) early days of OSS and the web.

The Blender community was and is an amazing combination of technology, creativity, and positivity, and I think we owe Ton for helping steward much of it.

You can do so by adding `https://data.commoncrawl.org/` instead of `s3://commoncrawl/` before each of the WARC/WAT/WET paths.

Comments URL: https://news.ycombinator.com/item?id=42217558

Points: 4

# Comments: 0

In 2016 my team from Salesforce Research published our work on the Quasi-Recurrent Neural Network[1] (QRNN). The QRNN variants we describe are near identical (minGRU) or highly similar (minLSTM) to the work here.

The QRNN was used, many years ago now, in the first version of Baidu's speech recognition system (Deep Voice [6]) and as part of Google's handwriting recognition system in Gboard[5] (2019).

Even if there are expressivity trade-offs when using parallelizable RNNs they've shown historically they can work well and are low resource and incredibly fast. Very few of the possibilities regarding distillation, hardware optimization, etc, have been explored.

Even if you need "exact" recall, various works have shown that even a single layer of attention with a parallelizable RNN can yield strong results. Distillation down to such a model is quite promising.

Other recent fast RNN variants such as the RWKV, S4, Mamba et al. include citations to QRNN (2016) and SRU (2017) for a richer history + better context.

The SRU work has also had additions in recent years (SRU++), doing well in speech recognition and LM tasks where they found similar speed benefits over Transformers.

I note this primarily as the more data points, especially when strongly relevant, the better positioned the research is. A number of the "new" findings from this paper have been previously explored - and do certainly show promise! This makes sure we're asking new questions with new insights (with all the benefit of additional research from ~8 years ago) versus missing the work from those earlier.

[1] QRNN paper: https://arxiv.org/abs/1611.01576

[2] SRU paper: https://arxiv.org/abs/1709.02755

[3]: SRU++ for speech recognition: https://arxiv.org/abs/2110.05571

[4]: SRU++ for language modeling: https://arxiv.org/abs/2102.12459

[5]: https://research.google/blog/rnn-based-handwriting-recogniti...

[6]: https://arxiv.org/abs/1702.07825

[1]: https://arxiv.org/abs/2108.12409

[2]: n.b. Ofir Press is co-creator of ALiBi https://twitter.com/OfirPress/status/1654538361447522305

WARC as a format essentially states that unless you have good reason "record at a time" compression is the preferred[1]. The mixture of "technically possible" and "part of spec" is what makes it so useful - any generic WARC tool can support random access, there are explicit fields to index over (URL), and even non-conforming WARC files can be easily rewritten to add such a capability.

[1]: https://iipc.github.io/warc-specifications/specifications/wa...

My favorite capability of the WARC format borrows from the fact that most compression formats can be written to allow random access. Compression formats such as `gzip` and `zstandard` allow multiple compressed streams to be stuck together and act during decompression as if it's one contiguous file.

Hence you can create multiple compressions and literally stick them together:

  $ echo cat > cat.txt
  $ echo dog > dog.txt
  $ zstd cat.txt dog.txt
  $ cat cat.txt.zst dog.txt.zst > catdog.zst
  $ zstdcat catdog.zst
  cat
  dog

Trading compression for random lookup is even more forgiving if you create a separate compression dictionary tailored toward your dataset. For web crawling you'd likely get you the majority of the compression gains back unless pages from the same website are sequentially written which is unlikely in most situations. The website's shared template/s would result in very high compression gains across files which you'd lose by allowing random lookup but most crawlers don't don't operate sequentially so local compression gains are likely smaller than larger.

[1]: https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requ...

The prices above are plus or minus from Lambda Labs[1].

[1]: https://lambdalabs.com/products/scalar

> Figma isn't using true CRDTs though. CRDTs are designed for decentralized systems where there is no single central authority to decide what the final state should be. There is some unavoidable performance and memory overhead with doing this. Since Figma is centralized (our server is the central authority), we can simplify our system by removing this extra overhead and benefit from a faster and leaner implementation.

> It’s also worth noting that Figma's data structure isn't a single CRDT. Instead it's inspired by multiple separate CRDTs and uses them in combination to create the final data structure that represents a Figma document.

https://www.figma.com/blog/how-figmas-multiplayer-technology...

The laptop is amazing when it's working, default Linux support too, but so far have literally had to ship it back to Dell for at least two months total since I've used it. Usually the turn around is only a few days but they were waiting on a part at one stage for weeks.

I am now looking to do a system exchange with them but am not certain if it's going to go through. At this stage I can't imagine they'd be profitable on this machine given how many times I've had to ship it in for repair.

Hence I sadly cannot recommend the XPS 13 even though I truly love it when it's working and laugh with glee at having 32GB of RAM =]

The author mentions in a tweet going from minutes to seconds for compression when switching CPU for GPU[2]. From memory he has made other references to a few seconds for compression being entirely reasonable for such tasks but I can't find a direct reference.

[1]: http://www.binomial.info/

[2]: https://twitter.com/richgel999/status/1476325003662667777

Given I had no clue what state the vehicles were in, and that they'd start moving without indication, it felt pretty damn worrying.

[1]: https://twitter.com/Smerity/status/1539144641206882304

Comments URL: https://news.ycombinator.com/item?id=31927830

Points: 2

# Comments: 0

For anyone who wanted more technical discussion re: ML / LM (though the author notes this work "[does] not reflect the architectures or latencies of my employer's models" i.e. it's an exploratory technical breakdown of general model characteristics) I've appreciated the technical write-ups from @kipperrii (ML ops @ Cohere) recently:

- Transformer Inference Arithmetic: https://carolchen.me/blog/transformer-inference-arithmetic/

- Breakdown of H100s for Transformer Inferencing: https://carolchen.me/blog/h100-inferencing/

Comments URL: https://news.ycombinator.com/item?id=30900295

Points: 2

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=30836233

Points: 6

# Comments: 0

I'd imagine most lost knowledge is not an explicit decision however which means such historical scenarios / documentation / ... are just lost as part of business. Lost knowledge is the default for companies.

Twitter is likely better than most given their documentation is all digital and there exist explicit processes to catalogue such incidents. I'd also be curious to see how much of this knowledge has been implicitly exported to their open source codebases.

Comments URL: https://news.ycombinator.com/item?id=30295542

Points: 138

# Comments: 26

> There are no charges for outbound data transfer when using a dedicated network connection to the RSTOR network and direct peering is established. These connections are charged separately. To find out more please contact us.

I would imagine for enterprise customers you might either be using direct peering / similar or they're able to charge based upon ballpark bandwidth usage. Otherwise I'm unsure as it does seem quite in line with Wasabi otherwise.

[1]: https://console.rstor.space/pricing

It's a shame as I love aspects of this type of storage service, even with the caveats, but they're not useful if there's no way to pay more for excess transfer. Luckily I think R2 fits that requirement (though paying for more operations vs paying for more transfer).

[1]: https://wasabi.com/paygo-pricing-faq/