Using Web Crawler in ServiceNow External Content Connectors

[Likitha Patel]

In many ServiceNow implementations, organizations require search results to include relevant content from trusted external websites. This article explains how to configure the Web Crawler capability in External Content Connectors to index external data and make it searchable within ServiceNow.

Overview

The Web Crawler connector enables ServiceNow to crawl external websites, extract content, and index it. This indexed content can be used in AI Search, Virtual Agent, Now Assist, and other AI-powered experiences.

Prerequisites

- Install the External Content Connectors – Web Crawler plugin (sn_ext_conn_wc).
- Ensure AI Search is configured in your instance.

Problem Statement

Agents often need information from external knowledge sources such as documentation sites. However, without integration, these sources are not searchable within ServiceNow. The goal is to enable unified search results that include both internal and external content.

Configuration Steps

1. Navigate to External Content Connectors > Web Crawler.
2. Click Create and select 'Web Crawler (Website)'.
3. Choose a predefined or custom web source.

1. Configure crawl settings such as frequency and search source.
2. Define crawl type (Full, Include URLs, Exclude URLs).
3. Create or select a Search Profile under AI Search.
4. Associate the crawler with the Search Profile.
5. Start the crawl and monitor progress in the admin home page.

Crawl Types

- Full Crawl: Crawls the entire website.
- Include URLs: Crawls only specified URLs.
- Exclude URLs: Crawls all except specified URLs.

Predefined Web Sources

ServiceNow provides predefined sources such as Apple Support, Microsoft, Google, and Atlassian. These help accelerate setup without manual configuration.

Limitations

1. URLs cannot be updated after connector creation. You need to delete the connector and crawl again.
2. A maximum of 3 custom web sources is allowed.
3. Each source can crawl up to 50,000 URLs or pages.

Conclusion

The Web Crawler feature enhances ServiceNow search by incorporating external knowledge sources. With proper configuration, organizations can deliver a unified and efficient search experience.

Thank you for reading.
If you found this helpful, please mark this article as helpful 👍