Comments URL: https://news.ycombinator.com/item?id=44322873

Points: 2

# Comments: 0

Overall, it will be some pain for sure. And to master it including PTX etc. will take a lot of time.

https://github.com/NX-AI/mlstm_kernels

https://huggingface.co/NX-AI/xLSTM-7b

1. Improving the Self-Attention in the Transformer as is, keeping the quadratic complexity, which has some theoretical advantage in principle[1]: The most hyped one probably DeepSeek's Multi-Latent Attention[15], which kind of is Attention still - but also somehow different.

2. Linear RNNs: This starts from Linear Attention[2], DeltaNet[3], RKWV[4], Retention[5], Gated Linear Attention[6], Mamba[7], Griffin[8], Based[9], xLSTM[10], TTT[11], Gated DeltaNet[12], Titans[13].

They all have an update like: C_{t} = F_{t} C_{t-1} + i_{t} k_{t} v_{t}^T with a cell state C and output h_{t} = C_{t}^T q_{t}. There's a few tricks that made these work and now being very strong competitors to Transformers. The key here is the combination of an linear associative memory (aka Hopfield Network, aka Fast Weight Programmer, aka State Expansion...) and pushing it into a sequence with gating similar to the original LSTM (input, forget, output gate) - while here this is only dependent on the current input not the previous state for linearity. The linearity is needed to make it sequence-parallelizable, there are efforts now to add non-linearities again, but let's see. Their main benefit+downside both is that they have a fixed-size state, and therefore linear (vs Transformer-quadratic) time complexity.

For larger sizes they have become popular in hybrids with Transformer (Attention) Blocks, as there are problems with long context tasks [14]. Cool thing is they can also be distilled from pre-trained Transformers with not too much performance drop [16].

3. Along the sequence dimension most things can be categorized in these two. Attention and Linear (Associative Memory Enhanced) RNNs are heavily using Matrix Multiplications and anything else would be a waste of FLOPs on current GPUs. The essence is how to store information and how to interact with it, there might be still interesting directions as other comments show. Other important topics that go into the depth / width of the model are: Mixture of Experts, Iteration (RNNs) in Depth[17].

Disclaimer: I'm author of xLSTM and we recently released a 7B model [18] trained at NXAI, currently the fastest linear RNN at this scale and performance. Happy to answer more questions on this or the current state in this field of research.

[1] https://arxiv.org/abs/2008.02217

[2] https://arxiv.org/abs/2006.16236

[3] https://arxiv.org/pdf/2102.11174

[4] https://github.com/BlinkDL/RWKV

[5] https://arxiv.org/abs/2307.08621

[6] https://arxiv.org/pdf/2312.00752

[7] https://arxiv.org/abs/2312.06635

[8] https://arxiv.org/pdf/2402.19427

[9] https://arxiv.org/abs/2402.18668

[10] https://arxiv.org/abs/2405.04517

[11] https://arxiv.org/abs/2407.04620

[12] https://arxiv.org/abs/2412.06464

[13] https://arxiv.org/abs/2501.00663

[14] https://arxiv.org/abs/2406.07887

[15] https://arxiv.org/abs/2405.04434

[16] https://arxiv.org/abs/2410.10254

[17] http://arxiv.org/abs/2502.05171

[18] https://huggingface.co/NX-AI/xLSTM-7b

For the matrix 'C' state, there are also heads/cells in that sense that you have multiple, but they don't talk to each other. So yes, you can view that as a 3D tensor. And here, the matrix is the fundamental building block / concept.

As a clarification: The speed for training will be on par with FlashAttention-2, when fully optimized and only including the mLSTM. For decoding/inference both are very close to Mamba as xLSTM is a recurrent architecture. The sLSTM has memory mixing, that is state tracking capabilities, for problems Transformers and State Space Models (and any other sequence-parallelizable architecture) cannot solve fundamentally.

Hall of Fame, Munich, Bavaria

Note that the point clouds might be a lot of data to download if enabled (~50 MB+). Currently limited to Bavaria, Germany.

Thanks for your feedback!

Comments URL: https://news.ycombinator.com/item?id=37853754

Points: 6

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=36707110

Points: 4

# Comments: 1

Comments URL: https://news.ycombinator.com/item?id=30198672

Points: 3

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=27531948

Points: 3

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=25650759

Points: 2

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=24217233

Points: 1

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=24103235

Points: 2

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=24102396

Points: 25

# Comments: 4