Creating multiple external content connectors of the same type
Summarize
Summary of Creating multiple external content connectors of the same type
ServiceNow AI Platform® allows administrators to create multiple external content connectors of the same type within a single instance. These connectors can crawl either different source systems or the same source system, including overlapping crawl scopes. This flexibility enables comprehensive content retrieval from various sites or sources while managing indexing and permissions effectively.
Show less
Supported Configurations
- Different source systems: Multiple connectors crawl distinct sets of sites across separate source systems without content duplication.
- Same source system with distinct scopes: Connectors target non-overlapping site groups in the same source system, avoiding duplication.
- Same source system with overlapping scopes: Connectors may share some sites, leading to potential retrieval of the same content from those sites by multiple connectors, but no duplication occurs from other sites.
Behavior Notes
- All connectors of the same type feed retrieved content into a single indexed source, enabling unified search results.
- During indexing, duplicate items retrieved by multiple connectors are deduplicated, ensuring only one copy appears in search results.
- User permissions are managed per connector; each connector requires its own user permission crawl to populate relevant security principals.
- Crawl jobs for connectors of the same type run in parallel if system resources permit; otherwise, they execute sequentially.
- The crawl limit of one million items applies individually to each connector, allowing significant content coverage across connectors.
Considerations for Parallel Execution
- Running multiple connectors crawling the same source system in parallel increases request load, which may trigger rate limits on the source system.
- Parallel crawl jobs writing to the shared indexed source may experience contention, potentially reducing retrieval rates.
Understanding these configurations and behaviors helps ServiceNow customers effectively plan and manage multiple external content connectors to optimize content ingestion, indexing, and security across diverse or overlapping data sources.
Admins can create multiple external content connectors of the same type on a single ServiceNow AI Platform® instance. These connectors can crawl different source systems or the same source system. Connectors that crawl the same source system can include overlapping crawl locations.
Supported configurations
- Multiple external content connectors of the same type that crawl different source systems
- For example, one Microsoft SharePoint Online external content connector crawls sites 1A and 1B in source system 1, while a second Microsoft SharePoint Online external content connector crawls sites 2A and 2B in source system 2. The two external content connectors have no sites in common, so there's no duplication of retrieved content.
- Multiple external content connectors of the same type that crawl the same source system with distinct crawl scopes
- For example, one Microsoft SharePoint Online external content connector crawls sites 1A and 1B in source system 1, while a second Microsoft SharePoint Online external content connector crawls sites 1C and 1D in source system 1. The two external content connectors have no sites in common, so there's no duplication of retrieved content.
- Multiple external content connectors of the same type that crawl the same source system with overlapping crawl scopes
- For example, one Microsoft SharePoint Online external content connector crawls sites 1A and 1B in source system 1, while a second Microsoft SharePoint Online external content connector crawls sites 1B and 1C in source system 1. The two external content connectors have site 1B in common, so content from that site may be retrieved by both connectors. There's no duplication of retrieved content from the other sites.
Behavior notes
- All external content connectors of the same type feed their retrieved content to the same indexed source
- In the previous examples, all content retrieved by either of the two Microsoft SharePoint Online external content connectors would be fed to the SharePoint Online indexed source.
- During indexing of crawled content, retrieved items are deduplicated across all external content connectors of the same type
- As an example, if two different Microsoft SharePoint Online external content connectors both retrieve the same file from the same site, only one copy of the retrieved file is indexed and appears in your search results.
- User permissions are specific to an individual external content connector, not shared between all external content connectors of the same type
- Admins need to run a separate user permission crawl for each external content connector of the same type. In the preceding examples, each Microsoft SharePoint Online external content connector needs its own user permission crawl. Running a user permission crawl for the first Microsoft SharePoint Online external content connector does not populate security principals for the second connector.
- The system executes crawl jobs for external content connectors of the same type in parallel if sufficient resources are available
- If sufficient resources for parallel execution aren't available, the system executes the crawl jobs sequentially instead.
- The crawl limit of one million items applies separately to each external content connector of the same type
- In the preceding examples, each individual Microsoft SharePoint Online external content connector can crawl up to one million items.
Considerations for parallel execution
- Increased request load on source system
- Creating multiple external content connectors that crawl the same source system in parallel can increase the rate at which requests are made to the source system. This may trigger rate-limit issues on the source system.
- Indexed source contention when running in parallel
- Running crawl jobs for multiple external content connectors of the same type in parallel can cause contention when they send retrieved content to their shared indexed source. This contention may reduce the connectors' retrieval rate when their crawl jobs are run in parallel.