May 31, 2024 · Our state space duality (SSD) framework allows us to design a new architecture (Mamba-2) whose core layer is an a refinement of Mamba's selective SSM that is 2-8X faster, while continuing to be competitive with Transformers on language modeling.

Mamba is a new state space model architecture showing promising performance on information-dense data such as language modeling, where previous subquadratic models fall short of Transformers.

May 31, 2024 · The Mamba-2 Architecture. Although the core contribution of Mamba-2 is the new SSD layer and theory, we also make some small changes to Mamba’s neural network architecture. The main change is producing the $(A, B, C)$ SSM parameters in parallel with the $X$ input, instead of sequentially.

Our state space duality (SSD) framework allows us to design a new architecture (Mamba-2) whose core layer is an a refinement of Mamba's selective SSM that is 2-8X faster, while continuing to be competitive with Transformers on language modeling.

Mamba 2 Overview. The Mamba2 model was proposed in Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality by Tri Dao and Albert Gu. It is a State Space Model similar to Mamba 1, with better performances in a simplified architecture. The abstract from the paper is the following:

Mamba is a new state space model architecture showing promising performance on information-dense data such as language modeling, where previous subquadratic models fall short of Transformers.

Jun 3, 2024 · Even though technically Mamba-2 is more restricted than Mamba-1 for the same N, the larger state dimensions generally improve model quality. Here we show results for models trained on 300B tokens on the Pile, with Mamba-2 outperforming Mamba-1 and Pythia.

Jun 10, 2024 · For example, they introduce Mamba-2, an architecture that refines and optimizes the selective state-space model (a variant of SSM) to be faster and more competitive with Transformers in...

Jun 5, 2024 · Mamba is a new approach to deep learning for sequences, built upon a flexible framework called Structured State Space Models (SSMs). You can think of SSMs as a general way to build sequence models, encompassing familiar architectures like RNNs and CNNs.

Jun 19, 2024 · Researchers Tri Dao and Albert Gu have unveiled Mamba-2, the next iteration of their widely popular Mamba-1 model on GitHub. This new model promises significant improvements and innovations in the realm of state space models, particularly for information-dense data like language models.