StarVector: Generating Scalable Vector Graphics Code from Images and Text

Abstract

Scalable Vector Graphics (SVGs) have become integral in modern image rendering and graphic design applications due to their infinite scalability in resolution, versatility, and editing capabilities. Traditional tasks related to SVG, such as image vectorization, have relied mainly on image processing algorithms. While these algorithms are proficient at replicating images accurately in many instances, they lack the semantic and visual comprehension required for more complex tasks, like text-conditioned SVG generation. Deep learning has made substantial progress in SVG modeling, but generating complex SVGs remains challenging. SVG syntax is rich, encompassing many commands such as lines, paths, polygons, text, or effects like color gradients. Previous methods can only use the path command, limiting the complexity of SVGs that could be created and making them challenging to interpret and manipulate. We present StarVector, a Multimodal Foundation Model for SVG generation that effectively integrates Code Generation Large Language Models (CodeLLMs) and Vision Models. StarVector can perform image vectorization and text-conditional SVG generation by learning to predict SVG codes from a sequence of visual (CLIP) or textual (LLM) representations. StarVector uses the full capacity of the SVG language. To build our foundation model, we introduce SVG-Stack, a large-scale dataset with over 2 million SVGs and textual captions, alongside SVG-Bench, a comprehensive benchmark for SVG generation. We demonstrate that our model outperforms previous methods on image vectorization, being the first to close the gap with traditional algorithms. Results on text-conditioned SVG generation show the ability to create complex and intricate designs, beating prior baselines in SVG-Stack and SVG-FIGR datasets. Code, models, and datasets will be open-sourced.