ServiceNow recherche

WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code Generation

Résumé

Understanding diverse web data and automating web development presents an exciting challenge for agentic AI. While existing benchmarks address isolated web-based tasks—such as website-based Visual Question Answering (VQA) and UI-to-code generation—they lack a unified evaluation suite for assessing web agents that interact with and reason about web environments. We introduce WebMMU, a large-scale benchmark for evaluating AI-driven web agents across multilingual website VQA, HTML/CSS/JavaScript code editing, and sketch-to-code generation. WebMMU provides a comprehensive evaluation suite with real-world website data, multi-step reasoning tasks, and functional UI understanding. Benchmarking state-of-the-art multimodal models on WebMMU reveals significant limitations in web-based reasoning, layout understanding, and structured code generation, particularly in preserving UI hierarchy, handling multilingual content, and producing robust, functional code. While most existing models are optimized for English-only settings, WebMMU highlights the challenges of cross-lingual adaptation in real-world web development. These findings expose critical gaps in current models’ ability to understand website structures, execute user instructions, and generate high-quality web code, underscoring the need for more advanced multimodal reasoning in AI-driven web understanding and development.

Publication
Workshop at the Computer Vision and Pattern Recognition Conference (CVPR)
Rabiul Awal
Rabiul Awal
Visiting Researcher

Visiting Researcher at AI Frontier Research located at Montreal, QC, Canada.

Aarash Feizi
Aarash Feizi
Visiting Researcher

Visiting Researcher at AI Frontier Research located at Montreal, QC, Canada.

Suyuchen Wang
Suyuchen Wang
Visiting Researcher

Visiting Researcher at AI Frontier Research located at Montreal, QC, Canada.

Christopher Pal
Christopher Pal
Distinguished Scientist

Distinguished Scientist at AI Research Partnerships & Ecosystem​ located at Montreal, QC, Canada.

David Vazquez
David Vazquez
Director of AI Research

Director of AI Research at AI Research Management located at Montreal, QC, Canada.

Siva Reddy
Siva Reddy
Research Scientist

Research Scientist at AI Research Partnerships & Ecosystem​ located at Montreal, QC, Canada.

Juan A. Rodriguez
Juan A. Rodriguez
Visiting Researcher

Visiting Researcher at AI Frontier Research located at Montreal, QC, Canada.

Perouz Taslakian
Perouz Taslakian
Research Lead

Research Lead at AI Frontier Research located at Montreal, QC, Canada.