ServiceNow Research

BigDocs: An Open and Permissively-Licensed Dataset for Training Multimodal Models on Document and Code Tasks

Abstract

Multimodal AI has the potential to significantly enhance document-understanding tasks, such as processing receipts, understanding workflows, extracting data from documents, and summarizing reports. Code generation tasks that require long-structured outputs can also be enhanced by multimodality. Despite this, their use in commercial applications is often limited due to limited access to relevant training data and restrictive licensing, which hinders open access. To address these limitations, we introduce BigDocs-7.5M, a high-quality, open-access dataset comprising 7.5 million multimodal documents across 30 tasks. We use an efficient data curation process to ensure that our data is high quality and license-permissive. Our process emphasizes accountability, responsibility, and transparency through filtering rules, traceable metadata, and careful content analysis. Additionally, we introduce BigDocs-Bench,, a benchmark suite with 10 novel tasks where we carefully create datasets that reflect real-world use cases involving reasoning over Graphical User Interfaces (GUI) and code generation from images. Our experiments show that training with BigDocs-Bench, improves average performance up to 25.8% over closed-source GPT-4o in document reasoning and structured output tasks such as Screenshot2HTML or Image2Latex generation. Finally, human evaluations revealed that participants preferred the outputs from models trained with BigDocs over those from GPT-4o. This suggests that BigDocs can help both academics and the open-source community utilize and improve AI tools to enhance multimodal capabilities and document reasoning.

Publication
International Conference of Learning Representations (ICLR)
Juan A. Rodriguez
Juan A. Rodriguez
Visiting Researcher

Visiting Researcher at AI Frontier Research located at Montreal, QC, Canada.

Tianyu Zhang
Tianyu Zhang
Visiting Researcher

Visiting Researcher at AI Frontier Research located at Montreal, QC, Canada.

Aarash Feizi
Aarash Feizi
Visiting Researcher

Visiting Researcher at AI Frontier Research located at Montreal, QC, Canada.

Abhay Puri
Abhay Puri
Applied Research Scientist

Applied Research Scientist at AI Research Deployment​ located at Montreal, QC, Canada.

Ahmed Masry
Ahmed Masry
Visiting Researcher

Visiting Researcher at AI Frontier Research located at Toronto, ON, Canada.

Shravan Nayak
Shravan Nayak
Visiting Researcher

Visiting Researcher at AI Frontier Research located at Montreal, QC, Canada.

Rabiul Awal
Rabiul Awal
Visiting Researcher

Visiting Researcher at AI Frontier Research located at Montreal, QC, Canada.

Amirhossein Abaskohi
Amirhossein Abaskohi
Visiting Researcher

Visiting Researcher at AI Frontier Research located at Vancouver, BC, Canada.

Suyuchen Wang
Suyuchen Wang
Visiting Researcher

Visiting Researcher at AI Frontier Research located at Montreal, QC, Canada.

Pierre-André Noël
Pierre-André Noël
Research Scientist

Research Scientist at AI Frontier Research located at Montreal, QC, Canada.

Torsten Scholak
Torsten Scholak
Research Lead

Research Lead at AI Research Deployment​ located at Montreal, QC, Canada.

Nicolas Chapados
Nicolas Chapados
VP of Research

VP of Research at AI Research Management located at Montreal, QC, Canada.

Sean Hughes
Sean Hughes
AI Ecosystem Director

AI Ecosystem Director at AI Research Partnerships & Ecosystem​ located at San Diego, CA, USA.

Siva Reddy
Siva Reddy
Research Scientist

Research Scientist at AI Research Partnerships & Ecosystem​ located at Montreal, QC, Canada.

Christopher Pal
Christopher Pal
Distinguished Scientist

Distinguished Scientist at AI Research Partnerships & Ecosystem​ located at Montreal, QC, Canada.

Issam H. Laradji
Issam H. Laradji
Research Manager

Research Manager at AI Frontier Research located at Vancouver, BC, Canada.

Spandana Gella
Spandana Gella
Research Manager

Research Manager at AI Frontier Research located at Montreal, QC, Canada.

Perouz Taslakian
Perouz Taslakian
Research Lead

Research Lead at AI Frontier Research located at Montreal, QC, Canada.

David Vazquez
David Vazquez
Director of AI Research

Director of AI Research at AI Research Management located at Montreal, QC, Canada.