Subscribe for updates

Abstract: Transformers have made a huge impact in sequence modelling, especially for vision and NLP tasks. But they do have a quadratic complexity with respect to the input sequence, which results in increasing complexity as resolution of images increases, for instance. State space models like S4 were initially proposed to bridge the complexity gap for longer sequences. Mamba is a popular SSM that addresses certain content awareness of other SSMs by incorporating selective scanning (S6) to capture long range dependencies with nearly linear computational complexity. Several hybrid models have been proposed in the recent past that combine the advantages of Mamba with transformers and bridge the performance gap that most SSMs have with state-of-art transformers.  This talk will give an outline of sequence modeling and how transformers impacted the same initially. Then it explains the SSMs and finally illustrates some of the hybrid models and how they work. It also gives some performance studies comparing the state-of-art transformers with hybrid models in areas of computer vision and time series forecasting on open source benchmarks.