ServiceNow Research

Azimuth: Systematic Error Analysis for Text Classification

Abstract

We present Azimuth, an open-source and easy-to-use tool to perform error analysis for text classification. Compared to other stages of the ML development cycle, such as model training and hyper-parameter tuning, the process and tooling for the error analysis stage are less mature. However, this stage is critical for the development of reliable and trustworthy AI systems. To make error analysis more systematic, we propose an approach comprising dataset analysis and model quality assessment, which Azimuth facilitates. We aim to help AI practitioners discover and address areas where the model does not generalize by leveraging and integrating a range of ML techniques, such as saliency maps, similarity, uncertainty, and behavioral analyses, all in one tool. Our code and documentation are available at github.com/servicenow/azimuth.

Publication
Conference on Empirical Methods in Natural Language Processing (EMNLP)
Gabrielle Gauthier Melançon
Gabrielle Gauthier Melançon
Applied Research Scientist

Applied Research Scientist at Azimuth located at Montreal, QC, Canada.

Orlando Marquez
Orlando Marquez
Applied Research Scientist

Applied Research Scientist at Azimuth located at Montreal, QC, Canada.

Lindsay Brin
Lindsay Brin
Applied Research Scientist

Applied Research Scientist at Azimuth located at Chelsea, QC, Canada.

Chris Tyler
Chris Tyler
AI Developer

AI Developer at Emerging Technologies Lab located at Montreal, QC, Canada.

Joseph Marinier
Joseph Marinier
Machine Learning Developer

Machine Learning Developer at Azimuth located at Montreal, QC, Canada.

Karine Grande
Karine Grande
Product Designer

Product Designer at Azimuth located at France.

Di Le
Di Le
AI/ML Design Strategist

AI/ML Design Strategist at AI Trust and Governance Lab located at San Diego, CA, USA.