Webcrawler external content connector

  • Release version: Yokohama
  • Updated April 21, 2026
  • 1 minute to read

    • The Webcrawler external content connector retrieves pages and subdomains from a public website and makes their content and metadata searchable in AI Search applications. This connector can crawl content from predefined public web sources or your own custom web sources.

    Note:
    This external content connector is not included in the External Content Connectors Application Suite application. To use this connector, you must install it separately. For details on installation, see Install External Content Connectors.

    The system automatically schedules monthly content crawls to retrieve updated content from pages and subdomains found on the selected website. Search administrators can run one-time content crawls to update content ahead of schedule. Content crawls feed their data to AI Search for indexing.

    The indexed content and metadata are stored as records in a connector-specific indexed source. Search administrators can create search sources from this indexed source and link them to search profiles to make the indexed records searchable in AI Search applications.

    Each Webcrawler connector can retrieve up to 50,000 items (URLs) from its source system when running content crawls.
    Note:
    This is an exception to the general content crawl limit of one million (1,000,000) items.

    By default, you can configure up to three Webcrawler connectors for custom web sources. If you need to retrieve items from more than three custom web sources, you can create a Customer Service and Support case at https://support.servicenow.com/now to request a limit increase for the Webcrawler connector.