Adaptive desktop actions for web-based tasks
Summarize
Summary of Adaptive desktop actions for web-based tasks
Adaptive desktop actions enable AI agents to automate repetitive tasks across web applications by interacting directly with browsers via an extension. This approach requires no preconfigured APIs, scripts, or backend logic, allowing automation through clicking, typing, and scrolling within web interfaces. This is particularly useful for automating workflows that involve multiple web applications without stable pre-set steps.
Show less
Key Features
- Operating Modes:
- Defined path: Executes fixed, preconfigured steps for stable workflows.
- Adaptive path: Plans and executes steps dynamically from a high-level goal, useful when workflows vary or cannot be fully defined in advance. Manual intervention is possible at any time.
- AI Model Provider: Uses AWS Anthropic Sonnet model for processing.
- User Roles:
- Administrators manage permissions and workflows.
- Developers build and configure AI agents.
- Fulfillers automate routine tasks.
- Requestors manage browser extensions and trigger automation.
- Interaction via Now Assist Panel: Users access desktop actions through an enhanced chat interface providing real-time status updates, periodic screenshots, and login prompts for external websites.
- Security and Privacy: Users can delete chat logs containing sensitive information after sessions.
- Limitations:
- Automation is limited to browser content only.
- No interaction with desktop applications or local files except downloading.
- Cannot upload local files; for such cases, defined desktop actions are recommended.
- Third-Party Website Access: Interactions with third-party services are direct between user and third party; customers must comply with those services’ terms and data policies.
- Beta Technology Notice: AI Desktop Actions rely on beta “computer use” technology from Anthropic and are provided “as is” without warranties. There are inherent risks including potential interruptions, errors, or data security concerns.
What This Enables for ServiceNow Customers
ServiceNow customers can automate complex, variable web-based workflows without needing custom backend integrations or scripts. Adaptive desktop actions empower users to streamline repetitive tasks involving multiple web applications efficiently. The ability to choose between defined and adaptive modes provides flexibility to handle both stable and dynamic processes. Integration with the Now Assist panel gives visibility and control during automation execution, including handling authentication for external sites. However, customers should be aware of current limitations regarding local file interactions and the beta status of the underlying AI technology.
Adaptive desktop actions enables AI agents to automate repetitive tasks across web applications through a browser extension. The agent interacts directly with the browser by clicking, typing, and scrolling, without preconfigured APIs, scripts, or back-end logic.
Adaptive desktop actions overview
Desktop actions are tools that AI agents use to interact with web applications through a browser extension.
When you configure an AI agent and select Desktop action as a tool, you choose how it operates:| Mode | How it works | Use when |
|---|---|---|
| Defined path | Follows fixed steps preconfigured in AI Desktop Actions. | The workflow is stable and steps are known in advance. |
| Adaptive path | Works from a high-level goal. Dynamically plans and executes steps based on your instructions. | The workflow varies or steps cannot be fully defined in advance. |
Specify a high-level goal — such as updating user roles or scheduling maintenance — and the agent plans and executes the steps to complete it. You can take manual control at any point.
How adaptive desktop actions work
LLM provider
Adaptive desktop actions use AWS Anthropic Sonnet model provider.
Users
| Users | Description |
|---|---|
| Administrators | Manage permissions, roles, and agentic workflows |
| Developers | Build and configure AI agents with desktop action tools |
| Fulfillers | Automate routine fulfillment tasks across multiple systems |
| Requestors | Manage browser extensions and submit requests that trigger AI agents to automate web workflows |
Operating desktop actions
You access desktop actions through the Now Assist panel that has enhanced chat enabled. The AI agent provides updates on its progress in the chat interface. As the agent works, you receive:
- Real-time status updates in the chat
- Periodic screenshots of the web pages the agent navigates
- Notifications when external websites require login credentials
When an external website requires login, you're prompted in the chat. Switch to the external website tab, provide your credentials, then switch back to the Now Assist panel. The agent continues after authentication is complete.
Limitation
Desktop actions operate as browser extensions with the following limitations:
- Can only access content within the browser
- Cannot interact with desktop applications or local files (except for downloading files)
- Cannot upload data from the local file system
For tasks requiring local file access, consider using defined desktop actions. For more information, see Defined path desktop actions for desktop and web-based tasks.
Third-party website access
When you use ServiceNow desktop actions to visit, access, log in to, or otherwise interact with (collectively, “Access”) websites, applications, or other digital properties owned or operated by a third party (“Third Party Services”), that Access is a direct interaction between you and the Third Party Services. You're responsible for adhering to any applicable terms and conditions, including all policies or statements governing the use of personal data, of the third party.
ServiceNow AI Desktop Actions rely on “computer use,” a beta technology provided by Anthropic. As a result, AI Desktop Actions users are subject to unique risks, including as described in Anthropic's documentation. Notwithstanding anything to the contrary in any customer agreement, or any other agreement governing a customer's use of ServiceNow offerings, ServiceNow AI Desktop Actions are provided “as is” and without representations or warranties of any kind, including any warranty that AI Desktop Actions will be uninterrupted, error free, or free of harmful components, or that any data, including customer data, will be secure or not otherwise lost or damaged.