ServiceNow Research

AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Understanding

Abstract

Aligning visual features with language embeddings is a key challenge in vision-language models (VLMs). The performance of such models hinges on having a good connector that maps visual features generated by a vision encoder to a shared embedding space with the LLM while preserving semantic similarity. Existing connectors, such as multilayer perceptrons (MLPs), often produce out-of-distribution or noisy inputs, leading to misalignment between the modalities. In this work, we propose a novel vision-text alignment method, AlignVLM, that maps visual features to a weighted average of LLM text embeddings. Our approach leverages the linguistic priors encoded by the LLM to ensure that visual features are mapped to regions of the space that the LLM can effectively interpret. AlignVLM is particularly effective for document understanding tasks, where scanned document images must be accurately mapped to their textual content. Our extensive experiments show that AlignVLM achieves state-of-the-art performance compared to prior alignment methods. We provide further analysis demonstrating improved vision-text feature alignment and robustness to noise.

Publication
Workshop at the International Conference of Learning Representation (ICLR)
Ahmed Masry
Ahmed Masry
Visiting Researcher

Visiting Researcher at AI Frontier Research located at Toronto, ON, Canada.

Juan A. Rodriguez
Juan A. Rodriguez
Visiting Researcher

Visiting Researcher at AI Frontier Research located at Montreal, QC, Canada.

Tianyu Zhang
Tianyu Zhang
Visiting Researcher

Visiting Researcher at AI Frontier Research located at Montreal, QC, Canada.

Suyuchen Wang
Suyuchen Wang
Visiting Researcher

Visiting Researcher at AI Frontier Research located at Montreal, QC, Canada.

Chao Wang
Chao Wang
Applied Research Scientist

Applied Research Scientist at AI Frontier Research located at Toronto, ON, Canada.

Aarash Feizi
Aarash Feizi
Visiting Researcher

Visiting Researcher at AI Frontier Research located at Montreal, QC, Canada.

Abhay Puri
Abhay Puri
Applied Research Scientist

Applied Research Scientist at AI Research Deployment​ located at Montreal, QC, Canada.

Pierre-André Noël
Pierre-André Noël
Research Scientist

Research Scientist at AI Frontier Research located at Montreal, QC, Canada.

Nicolas Chapados
Nicolas Chapados
VP of Research

VP of Research at AI Research Management located at Montreal, QC, Canada.

Christopher Pal
Christopher Pal
Distinguished Scientist

Distinguished Scientist at AI Research Partnerships & Ecosystem​ located at Montreal, QC, Canada.

Issam H. Laradji
Issam H. Laradji
Research Manager

Research Manager at AI Frontier Research located at Vancouver, BC, Canada.

David Vazquez
David Vazquez
Director of AI Research

Director of AI Research at AI Research Management located at Montreal, QC, Canada.

Perouz Taslakian
Perouz Taslakian
Research Lead

Research Lead at AI Frontier Research located at Montreal, QC, Canada.

Spandana Gella
Spandana Gella
Research Manager

Research Manager at AI Frontier Research located at Montreal, QC, Canada.