StarFlow: Using AI to turn sketches into workflows

Today's business environment moves fast. Workflow automation has become essential for streamlining operations, reducing manual tasks, and ensuring processes run smoothly. Yet designing workflows can feel overly complicated. Although low-code platforms have made workflow design easier, the process still requires a deep understanding of system logic and execution rules.

Imagine having an application that could turn your business process sketch from a whiteboard brainstorming session or a project proposal into a working workflow instantly and effortlessly. That’s the vision behind StarFlow, our latest innovation, which uses vision-language models (VLMs) to bridge the gap between conceptualization and implementation.

What is StarFlow?

StarFlow is an AI-powered framework that helps convert hand-drawn sketches and digital diagrams into structured workflows. Relying on VLMs, StarFlow understands visual diagrams and turns them into executable workflows—without any manual coding—in three steps:

1. Sketch your workflow on paper, a whiteboard, or digitally with a tool such as Lucidchart.
2. Let AI analyze the key workflow elements—triggers, actions, conditions, and logic—directly from your sketch to create a structured JSON file.
3. Run your workflow in ServiceNow Flow Designer.

For example, a hand-drawn diagram illustrating an employee onboarding process—with steps such as document verification, account setup, and email notifications—can be automatically converted into a digital workflow, with no coding required (see Figure 1).

Figure 1: Given a sketch of a workflow, StarFlow analyzes the business process depicted in the image and converts it into JSON format that can be readily run in Flow Designer.

Overcoming limitations of AI models and datasets

Despite the impressive advancements in AI, existing models fall short when generating structured workflows from sketches due to:

• Ambiguity: Handwritten text, varying diagram styles, and inconsistent symbols are difficult for AI models to interpret.
• Missing industry knowledge: General-purpose AI models often fail to recognize business-specific workflow elements, leading to inaccuracies.
• Complex logic: Extracting precise execution rules from a static diagram requires advanced reasoning capabilities.

To address these limitations, we trained StarFlow on a diverse dataset of real-world, synthetic, and human-annotated workflow diagrams. By fine-tuning open-weight VLMs, we significantly improved the accuracy of structured workflow generation, outperforming traditional VLMs on this specialized task (see Figure 2).

Figure 2: To generate our dataset, we start from preexisting workflows and convert them to diagrams automatically. We then ask human labelers to convert the diagrams into manual or digital sketches.

Why StarFlow matters

StarFlow has the potential to revolutionize workflow automation across multiple industries in four key ways:

1. Create workflows fast: There’s no need for manual configuration. Simply sketch a process and let our model do the heavy lifting. This reduces the time required to build complex workflows, accelerating business automation initiatives.

2. Empower your entire team: Nontechnical users can create workflows using intuitive visual representations. This reduces dependency on IT teams, allowing for greater agility in business process management.

3. Boost accuracy and consistency: AI-powered analysis ensures consistency in workflow structures. It also minimizes errors caused by manual configuration.

4. Integrate cross-platform workflows: Easily facilitate the migration of workflows between enterprise automation platforms. This helps businesses integrate AI-driven automation seamlessly across their existing tools.

What's next for StarFlow

While StarFlow is a significant step toward making workflow automation more accessible, future advancements will focus on:

• Enhancing model generalization for different sketching styles and diagram formats
• Improving context-aware workflow generation using retrieval-augmented generation
• Expanding support for multimodal inputs, such as a combination of images and text

By harnessing the power of AI and VLMs, StarFlow is transforming the way businesses approach workflow automation—making it faster, smarter, and more intuitive. Stay tuned as we continue to push the boundaries of AI-driven automation.

Read our paper: https://arxiv.org/abs/2503.21889.

Then find out more about ServiceNow Research.