Transformer-SSM hybrids

Apriel-SSM: Converting Pre-Trained Transformer LLMs Into Subquadratic Hybrid Models Through Iterative End-to-End Distillation

Apriel-SSM: Converting Pre-Trained Transformer LLMs Into Subquadratic Hybrid Models Through Iterative End-to-End Distillation

Large Language Models achieve their success through transformer architectures with attention mechanisms that compute token …

Oleksiy Ostapenko, Shambhavi Mishra, Luke Kumar, Denis Kocetkov, Raymond Li, Joel Lamy Poirier, Sébastien Paquet, Torsten Scholak

NOW AI, 2025.