ServiceNow AI Research

Code Generation

GitChameleon: Evaluating AI Code Generation Against Python Library Version Incompatibilities
GitChameleon: Evaluating AI Code Generation Against Python Library Version Incompatibilities
The rapid evolution of software libraries presents a significant challenge for code generation models, which must adapt to frequent …
Nizar Islah, Justine Gehring, Diganta Misra, Eilif Muller, Irina Rish, Eilif Benjamin Muller, Massimo Caccia
Annual Meeting of the Association for Computational Linguistics (ACL),  2026.
Paper Cite Code
StarFlow: Generating Structured Workflow Outputs From Sketch Images
StarFlow: Generating Structured Workflow Outputs From Sketch Images
Workflows are a fundamental component of automation in enterprise platforms, enabling the orchestration of tasks, data processing, and …
Patrice Béchard, Chao Wang, Juan A. Rodriguez, Amirhossein Abaskohi, Christopher Pal, David Vazquez, Spandana Gella, Sai Rajeswar Mudumba, Perouz Taslakian
European Chapter of the Association for Computational Linguistics (EACL),  2026.
Paper Cite Code Video
GitChameleon: Evaluating AI Code Generation Against Python Library Version Incompatibilities
GitChameleon: Evaluating AI Code Generation Against Python Library Version Incompatibilities
The rapid evolution of software libraries presents a significant challenge for code generation models, which must adapt to frequent …
Nizar Islah, Justine Gehring, Diganta Misra, Eilif Muller, Irina Rish, Eilif Benjamin Muller, Massimo Caccia
Workshop at the Neural Information Processing Systems (NeurIPS),  2025.
Paper Cite Code
WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code Generation
WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code Generation
Understanding diverse web data and automating web development presents an exciting challenge for agentic AI. While existing benchmarks …
Rabiul Awal, Mahsa Massoud, Zichao Li, Aarash Feizi, Suyuchen Wang, Christopher Pal, Aishwarya Agrawal, David Vazquez, Siva Reddy, Juan A. Rodriguez, Perouz Taslakian, Sai Rajeswar Mudumba
Workshop at the Computer Vision and Pattern Recognition Conference (CVPR),  2025.
Paper Cite
WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code Generation
WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code Generation
Understanding diverse web data and automating web development presents an exciting challenge for agentic AI. While existing benchmarks …
Rabiul Awal, Mahsa Massoud, Zichao Li, Aarash Feizi, Suyuchen Wang, Christopher Pal, Aishwarya Agrawal, David Vazquez, Siva Reddy, Juan A. Rodriguez, Perouz Taslakian, Spandana Gella, Sai Rajeswar Mudumba
Workshop at the International Conference of Learning Representation (ICLR),  2025.
Paper Cite
BigDocs: An Open and Permissively-Licensed Dataset for Training Multimodal Models on Document and Code Tasks
BigDocs: An Open and Permissively-Licensed Dataset for Training Multimodal Models on Document and Code Tasks
Multimodal AI has the potential to significantly enhance document-understanding tasks, such as processing receipts, understanding …
Juan A. Rodriguez, Xiangru Jian, Siba Smarak Panigrahi, Tianyu Zhang, Aarash Feizi, Abhay Puri, Akshay Kalkunte, Francois Savard, Ahmed Masry, Shravan Nayak, Rabiul Awal, Mahsa Massoud, Amirhossein Abaskohi, Zichao Li, Suyuchen Wang, Pierre-André Noël, Mats L. Richter, Saverio Vadacchino, Shubham Agarwal, Sanket Biswas, Sara Shanian, Ying Zhang, Sathwik Tejaswi Madhusudhan, João Monteiro, Krishnamurthy (Dj) Dvijotham, Torsten Scholak, Nicolas Chapados, Sepideh Kharaghani, Sean Hughes, Tamer Özsu, Siva Reddy, Marco Pedersoli, Yoshua Bengio, Christopher Pal, Issam H. Laradji, Spandana Gella, Perouz Taslakian, David Vazquez, Sai Rajeswar Mudumba
International Conference of Learning Representations (ICLR),  2025.
Paper Cite Code Video
Evaluating In-Context Learning of Libraries for Code Generation
Evaluating In-Context Learning of Libraries for Code Generation
Contemporary Large Language Models (LLMs) exhibit a high degree of code generation and comprehension capability. A particularly …
Arkil Patel, Siva Reddy, Dzmitry Bahdanau, Pradeep Dasigi
North American Chapter of the Association for Computational Linguistics (NAACL),  2024.
Paper Cite Code
InCoRo: In-Context Learning for Robotics Control with Feedback Loops
InCoRo: In-Context Learning for Robotics Control with Feedback Loops
One of the challenges in robotics is to enable robotic units with the reasoning capability that would be robust enough to execute …
Jiaquiang Ye Zhu, Carla Gomez, David Vazquez, Michal Drozdzal
ArXiv,  2024.
Paper Cite
StarCoder 2 and The Stack v2: The Next Generation
StarCoder 2 and The Stack v2: The Next Generation
The BigCode project, an open-scientific collaboration focused on the responsible development of Large Language Models for Code (Code …
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Terry Yue Zhuo, Nii Osae Osae Dade, Lucas Krauß, Naman Jain, Yixuan Su, Xuanli He, Edoardo Abati, Yekun Chai, Xiangru Tang, Christopher Akiki, Chenghao Mou, Binyuan Hui, Nicolas Patry, Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sébastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Patwary, Nima Tajbakhsh, Yacine Jernite, Carlos Muñoz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf , Arjun Guha, Leandro von Werra, Harm de Vries, Alex Gu, Armel Zebaze, Evgenii Zheltonozhskii, Jian Zhu, Manan Dey, Marc Marone, Mayank Mishra, Muhtasham Oblokulov, Olivier Dehaene, Qian Liu, Tri Dao, Wenhao Yu, Niklas Muennighoff
ArXiv,  2024.
Paper Cite Code Video
StarVector: Generating Scalable Vector Graphics Code from Images and Text
StarVector: Generating Scalable Vector Graphics Code from Images and Text
Scalable Vector Graphics (SVGs) have become integral in modern image rendering and graphic design applications due to their infinite …
Juan A. Rodriguez, Shubham Agarwal, Abhay Puri, Issam H. Laradji, Sai Rajeswar Mudumba, Pau Rodriguez, David Vazquez, Christopher Pal, Marco Pedersoli
ArXiv,  2024.
Paper Cite