Defined desktop actions for desktop and web-based tasks
Automate desktop and web-based tasks that involve fixed steps using AI Desktop Actions.
AI Desktop Actions is a client application that is installed on the Windows operating system. The app offers two workspaces, the Design workspace, where you create and configure desktop automations, and the Execution workspace, where those automations run. Design workspace enables you to automate multi-step processes by recording with AI or manually capturing steps. Execution workspace enables AI agents to execute desktop actions in an isolated desktop session.
- On-screen task: These desktop actions help you simulate humans interacting with UI elements on your thick client applications, legacy systems, or SaaS applications without APIs. These actions include clicking buttons, typing into text boxes, selecting from dropdown menus, and more.
- Background task: These desktop actions include prebuilt connectors that enable your AI agents to interact with various applications and system components in the background.
Key interfaces related to AI Desktop Actions
- Design workspace
- Create, manage, and test desktop actions that define how automations interact with desktop and web applications. You can create desktop actions by recording with AI or manually capturing steps.
- Execution workspace
- Automatically runs desktop actions in an isolated desktop session during testing or execution. You don't open this workspace directly.
- AI Agent Studio
- Create, manage, and test AI agents that run desktop actions.
- Now Assist panel
- Trigger desktop actions from within ServiceNow.
These components work together to separate design, execution, and monitoring, ensuring secure and reliable desktop automation.
AI Desktop Actions users
| User | Description |
|---|---|
| Administrators |
Set up and deploy the AI Desktop Actions application. |
| Developers | Design, manage, and deploy automation solutions within their organization. |
| Business users | Automate repetitive desktop tasks without extensive technical knowledge. |
| Process owners | Optimize business processes by automating manual desktop interactions. |
How AI Desktop Actions works end-to-end
This workflow shows how AI Desktop Actions works end to end from designing desktop actions to running them during execution. As you move through the workflow, note which steps you perform manually and which steps are handled automatically by the system.
- When users with the AI Agent Admin (sn_aia.admin) role sign in from their desktop, they can access the Home page and Design workspace to create desktop actions. When they test a desktop action, the Execution workspace launches automatically.
- When users with the Now Assist panel user (now_assist_panel_user) role trigger an automation from the Now Assist panel, the Execution workspace launches automatically to run the desktop action.
You don’t open the Execution workspace directly. It launches automatically when you test or run a desktop action.
- With the AI Agent Admin role, use the no-code Design workspace to create reusable desktop actions that AI agents can execute. You can visually create and simulate application interactions, such as clicks, text entry, and extraction that mimic human behavior using screen and UI element recognition.
- When you test a desktop action, the system automatically launches the Execution workspace in an isolated desktop session. You do not open or manage the Execution workspace directly.
- Activate your desktop actions from the Design workspace so that AI agents can use them as tools in AI Agent Studio.
- Create AI agents by providing simple text instructions in AI Agent Studio.
- Add your designed desktop actions as tools to your AI agents.
- Verify your AI agents function correctly with their integrated tools.
- With the Now Assist panel user role, provide instructions for the task you want to complete on your desktop in the Now Assist panel.
- An AI agent is triggered from the Now Assist panel to execute the desktop actions.
- The AI agent selects the appropriate desktop action to complete the task.
- The system automatically launches the Execution workspace in an isolated desktop session to run the desktop actions. You do not open or manage the Execution workspace directly.
- Observe and manage automations as they run in real time in the Execution workspace, which provides a dedicated execution environment.
- Step in and out of the automation where your inputs are required.
- Check the outcome of the execution in the Now Assist panel.
For more information, see Defined path desktop actions in AI Desktop Actions and Creating AI agents for AI Desktop Actions.
AI Desktop Actions capabilities
- Design desktop actions
-
- Create, configure, and manage reusable desktop actions with application metadata.
- With the Recorder feature, automatically record and capture user interactions and contextual information while you perform actions on desktop applications. You can record with AI for accurate anchor insertion and generating screen context.
- Additionally, manually capture screens, define UI interactions, and structure steps in a no-code Design workspace.
- Run in the background
- Enable silent and non-intrusive execution of agent tasks, supporting uninterrupted user workflows.
- Support for diverse applications on Windows
- Automate tasks across a wide variety of environments and UI types, including legacy systems, thick client applications, and web applications on Windows operating system.
- Core desktop capabilities
- Support common desktop actions such as form filling, application clicks, and OS file handling.
- Performance
- Supports automations with native integration to AI Agent Studio.
- Seamless integration
- Publish configured desktop actions directly to AI Agent Studio for easy access and deployment.