Action recorder in AI Desktop Actions
Summarize
Summary of Action recorder in AI Desktop Actions
The Action recorder in AI Desktop Actions enables you to capture and automate repetitive desktop tasks by recording your interactions with application elements. It saves these steps as reusable desktop actions, displayed as screenshots with annotated anchors and steps in the Design workspace. This tool helps you build automated workflows by replicating your actions on desktop applications.
Show less
Key Features
- Record with AI (Recommended): Uses AI to generate accurate anchor positions, validate and correct them post-recording, and create descriptive screen contexts to clarify each screen’s function. This reduces manual adjustments and increases automation reliability. Requires the ServiceNow AI Lens skill and the
sndesktopcore.desktopactionuserrole. - Auto capture with recorder: Automatically records interactions and adds anchors and steps without AI processing.
- Manual capture: Allows manual screen capture without automatic step recording, giving full control over the process.
- Recorder controls: Options to start, pause, restart, discard, or stop recordings, with each UI action logged sequentially (e.g., mouse clicks).
Best Practices for Accurate Recording
- Ensure the highlighter is visible on the target control before performing actions.
- Avoid rapid actions; wait for the recorder to finish processing each step (indicated by color changes in the recorder panel).
- Review and recapture steps involving hover-based components or potentially inaccurate anchors.
- Click check boxes and radio buttons near the center to avoid misfires.
- Perform scrolling in small increments to ensure each scroll is accurately captured.
- Verify correct anchor selection for desktop icons, especially when multiple user sessions can affect layouts.
- If screen latency causes incorrect anchors, recapture the affected screen.
How Screens and Steps Are Captured
Steps are grouped into screens to preserve context. New screens are recorded when:
- Windows change
- Buttons or options that alter layout are clicked
- Pages are scrolled
- Special keyboard keys like Enter or Tab are used
This ensures accurate representation of user interactions for playback.
Limitations and Capacity
- Scrolls performed via laptop touchpads are not captured.
- A maximum of 50 steps can be recorded per session; additional steps require starting a new session, which appends to previous recordings.
Practical Benefits for ServiceNow Customers
Using the Action recorder, especially with AI-enabled recording, allows ServiceNow customers to efficiently automate complex desktop workflows with higher accuracy and less manual effort. This leads to more reliable automation, faster setup times, and clearer documentation of automated steps through generated screen contexts and anchors. Ensuring proper permissions and AI Lens activation is key to leveraging these capabilities fully.
With Action recorder, you can capture steps to automate repetitive tasks in AI Desktop Actions. You can save the steps that you perform on application elements as a reusable desktop action.
Action recorder helps you record your interactions with desktop applications to create automated workflows. By recording steps, you can automate tasks that replicate your interactions. The recorded actions are displayed as screenshots in the Design workspace with anchors and steps automatically added.
The recorder captures the following items:
- Visual snapshots of your screen at key interaction points.
- Steps in the form of the buttons, fields, and other interface components you interact with.
Record with AI
Record with AI is the recommended way to create desktop actions. Record with AI generates more accurate anchor positions automatically, reducing the time you spend on manual anchor adjustments.
When you record with AI, after you finish recording, AI analyzes the recording, validates anchor positions, and corrects inaccuracies before you save or activate the action. AI also generates a screen context for each captured screen and description for the desktop action. Screen context is a description of what the screen does and what it contains, which helps reviewers and AI agents understand the screen's intent.
After AI processing completes, a confirmation banner appears: AI analysis complete. Verify AI generated anchors and screen contexts before continuing. Review and refine the AI-generated anchors and screen contexts before activating the action.
Capture modes
You can select a capture mode from the Capture options menu in the Screens and Steps panel.
- Record with AI (recommended)
- Records your interactions by capturing screens and adding steps and uses AI to automatically validate anchor positions and generate screen contexts at design time, reducing the risk of automation failures at runtime.
- Auto capture with recorder
- Records your interactions automatically and adds anchors and steps without AI processing.
- Manual capture
- Captures screens without automated recording, giving you full control over what is captured.
Recorder controls
You can select any of the following options from the More options menu on the Action recorder panel:
- Start recording: Start recording the steps
- Pause: Skip recording steps
- Restart: Restart recording the steps
- Discard: Discard the recording if it doesn't meet your needs
- Stop recording: Stop recording steps
Each step that you perform is captured sequentially and the type of UI action is displayed for each step. For example, Capturing Mouse Left Click event.
Limitations
Scrolls performed using a laptop touchpad aren't captured.
Tips for accurate recording
Follow these tips to improve the accuracy of your recordings.
- Before performing any action, ensure the highlighter is visible on the target control. Recording too quickly may prevent the recorder from capturing the action sequence correctly.
- Avoid performing steps too quickly. When the message in the Action recorder panel appears in red, wait for the recorder to finish processing the current step before continuing.
Proceed with the next step when the red highlight box appears around the element you interact with and the message in the Action recorder panel changes to blue.
- Temporarily hovering components, such as drop-downs, multi-select fields, reference fields, and date pickers may not always be captured during recording, or may generate inaccurate anchors. Review the recording after completion and recapture any affected steps as needed.
- When selecting check boxes or radio buttons, click as close to the center of the control as possible to avoid activating areas outside the intended element.
- Scrolling the page or hovering over a field may result in inaccurate anchors. Avoid both while recording.
- Perform scrolls in small increments, one scroll at a time, so the recorder captures each scroll action accurately.
- On the desktop home screen, the system may select the Windows icon instead of the Chrome icon as the anchor due to differences in the desktop layout between the main and inset user sessions. Verify that the correct icon is selected after recording.
- In rare instances, latency between capturing a screenshot and retrieving control details may cause an anchor to reflect an element from a subsequent screen rather than the intended one. If this occurs, recapture the affected screen.
How screens are captured
- Window is changed
- Button is clicked or option is selected that might change the layout
- Page is scrolled
- Special keyboard keys are used, such as Enter or Tab
You can capture a maximum of 50 steps using the recorder in a recording session. While auto-capturing steps, a counter displays the remaining number of steps you can capture (for example, "35 of 50 max"). Recording stops automatically after you capture 50 steps. If you need to capture additional steps, start a new recording session. The new recording adds screens and steps to those captured in the previous recording.