[CVPR 2024] VideoMamba: Overcoming the Quadratic Bottleneck in Video Understanding

TL;DR

Transitioning from images to videos introduces a massive leap in data redundancy and sequence length. VideoMamba replaces the traditional attention mechanism with a Selective State Space Model (SSM), achieving linear complexity ($O(L)$). It processes long videos 6x faster and uses 40x less memory than Transformers, while beating heavyweights like ViViT and TimeSformer on standard benchmarks.

Problem: The "Long Video" Memory Wall

Video understanding requires capturing both local motion and global context. Current architectures fall into two traps:

1. 3D CNNs: Excellent at local textures but "blind" to long-range dependencies.
2. Video Transformers: While great at global modeling, their self-attention mechanism is quadratic. Doubling the video length quadruples the memory cost, making "hour-long" video understanding an engineering nightmare on current hardware.

The authors ask: Can we have the global reach of Transformers with the efficiency of CNNs? The answer lies in the Mamba architecture—a selective SSM that has already revolutionized long-context NLP.

Methodology: Spatiotemporal Selective Scanning

VideoMamba adapts the Mamba block (originally 1D) for 3D video volumes. The core innovation is how the model "sees" the video patches.

1. The Bidirectional Mamba (B-Mamba)

Unlike text, video is not just "forward-moving"; spatial relationships are omnidirectional. VideoMamba employs a bidirectional scan to ensure the hidden state captures information from both the past/future frames and surrounding pixels.

2. Spatial-First Scanning

The researchers tested multiple ways to flatten 3D video tokens into 1D sequences for the SSM. They discovered that Spatial-First (scanning all pixels in a frame before moving to the next) is the most effective. This allows the model to utilize weights pretrained on 2D image datasets (like ImageNet) more effectively.

Figure 1: The VideoMamba framework follows a ViT-style isotropic design but replaces Multi-Head Latent Attention with B-Mamba blocks.

3. Solving the Scaling Curse

A known issue with pure SSMs in vision is overfitting as the model size grows (Mamba-Base often performs worse than Mamba-Small). VideoMamba introduces Self-Distillation: using a well-trained smaller model to guide the larger one, ensuring stable convergence during scaling.

Experiments: Efficiency Meets Accuracy

The performance gains of VideoMamba are most visible in two areas: extreme efficiency and long-term reasoning.

Efficiency Benchmark

For a 64-frame video, VideoMamba handles the sequence with ease, while TimeSformer hits a memory wall.

• Inference Speed: 6x faster than TimeSformer.
• GPU Memory: 40x less consumption (demonstrated in the paper's Figure 1).

Performance on Benchmarks

• Kinetics-400 (Short-term): VideoMamba-M (74M params) hits 83.3% Top-1 Accuracy, outperforming ViViT-L (311M params).
• Breakfast (Long-term): For complex procedural tasks, it achieves 97.9%, a significant jump over prior feature-based methods like ViS4mer.

Figure 2: GPU Memory and Speed trade-offs. VideoMamba stays nearly flat while Transformers' costs explode with sequence length.

Critical Insight: Why Mamba for Video?

The "Selective" part of the Selective State Space Model is key. Unlike previous SSMs (like S4) which were linear time-invariant, Mamba's S6 mechanism allows the model to decide which information to keep or forget based on the current input. This is perfect for video, where many frames are redundant (the background stays the same), but a single moment of "motion" is critical.

Conclusion & Limitations

VideoMamba presents a compelling case for the post-Transformer era in computer vision. It achieves what was previously thought impossible: Transformer-level accuracy with linear-time efficiency.

Limitations: The authors acknowledge that they haven't yet scaled to "billion-parameter" levels (VideoMamba-g) or fully integrated with Large Language Models (LLMs). However, as a backbone, VideoMamba is arguably the most efficient engine currently available for the next generation of video-AI.

Takeaway for Practitioners: If you are working on long-form video analysis (surveillance, movies, sports) and are limited by VRAM, VideoMamba is the architecture to watch in 2024.