Creating multiple external content connectors of the same type
Summarize
Summary of Creating multiple external content connectors of the same type
ServiceNow AI Platform® allows admins to create multiple external content connectors of the same type within a single instance. These connectors can crawl different source systems or the same source system, including overlapping crawl locations. This capability provides flexibility in how content is aggregated and indexed from external sources.
Show less
Supported Configurations
- Different source systems: Multiple connectors of the same type can independently crawl separate source systems without content duplication (e.g., SharePoint sites 1A/1B on system 1 and 2A/2B on system 2).
- Same source system with distinct scopes: Connectors can crawl different, non-overlapping areas within the same source system (e.g., SharePoint sites 1A/1B and 1C/1D on the same system) without duplicated content.
- Same source system with overlapping scopes: Connectors may crawl overlapping parts of the same source system (e.g., both crawling site 1B); retrieved content from the overlapping areas may be duplicated across connectors but will be deduplicated during indexing.
Behavior Notes
- All connectors of the same type feed retrieved content into a single shared indexed source, enabling unified search results.
- Content retrieved by multiple connectors is deduplicated during indexing, so identical files appear only once in search results.
- User permissions are managed per individual connector; each connector requires a separate user permission crawl to populate security principals.
- Crawl jobs for connectors of the same type can run in parallel if resources permit; otherwise, they run sequentially.
- The crawl limit of one million items applies separately to each connector, allowing each to crawl up to one million items independently.
Considerations for Parallel Execution
- Running multiple connectors concurrently against the same source system can increase request load, potentially triggering rate-limit issues on that system.
- Parallel crawl jobs feeding the same indexed source can cause contention, possibly reducing retrieval rates.
By understanding these configurations and behaviors, ServiceNow customers can effectively deploy multiple external content connectors of the same type to optimize content crawling, indexing, and search performance while managing system resource use and source system constraints.
Admins can create multiple external content connectors of the same type on a single ServiceNow AI Platform® instance. These connectors can crawl different source systems or the same source system. Connectors that crawl the same source system can include overlapping crawl locations.
Supported configurations
- Multiple external content connectors of the same type that crawl different source systems
- For example, one Microsoft SharePoint Online external content connector crawls sites 1A and 1B in source system 1, while a second Microsoft SharePoint Online external content connector crawls sites 2A and 2B in source system 2. The two external content connectors have no sites in common, so there's no duplication of retrieved content.
- Multiple external content connectors of the same type that crawl the same source system with distinct crawl scopes
- For example, one Microsoft SharePoint Online external content connector crawls sites 1A and 1B in source system 1, while a second Microsoft SharePoint Online external content connector crawls sites 1C and 1D in source system 1. The two external content connectors have no sites in common, so there's no duplication of retrieved content.
- Multiple external content connectors of the same type that crawl the same source system with overlapping crawl scopes
- For example, one Microsoft SharePoint Online external content connector crawls sites 1A and 1B in source system 1, while a second Microsoft SharePoint Online external content connector crawls sites 1B and 1C in source system 1. The two external content connectors have site 1B in common, so content from that site may be retrieved by both connectors. There's no duplication of retrieved content from the other sites.
Behavior notes
- All external content connectors of the same type feed their retrieved content to the same indexed source
- In the previous examples, all content retrieved by either of the two Microsoft SharePoint Online external content connectors would be fed to the SharePoint Online indexed source.
- During indexing of crawled content, retrieved items are deduplicated across all external content connectors of the same type
- As an example, if two different Microsoft SharePoint Online external content connectors both retrieve the same file from the same site, only one copy of the retrieved file is indexed and appears in your search results.
- User permissions are specific to an individual external content connector, not shared between all external content connectors of the same type
- Admins need to run a separate user permission crawl for each external content connector of the same type. In the preceding examples, each Microsoft SharePoint Online external content connector needs its own user permission crawl. Running a user permission crawl for the first Microsoft SharePoint Online external content connector does not populate security principals for the second connector.
- The system executes crawl jobs for external content connectors of the same type in parallel if sufficient resources are available
- If sufficient resources for parallel execution aren't available, the system executes the crawl jobs sequentially instead.
- The crawl limit of one million items applies separately to each external content connector of the same type
- In the preceding examples, each individual Microsoft SharePoint Online external content connector can crawl up to one million items.
Considerations for parallel execution
- Increased request load on source system
- Creating multiple external content connectors that crawl the same source system in parallel can increase the rate at which requests are made to the source system. This may trigger rate-limit issues on the source system.
- Indexed source contention when running in parallel
- Running crawl jobs for multiple external content connectors of the same type in parallel can cause contention when they send retrieved content to their shared indexed source. This contention may reduce the connectors' retrieval rate when their crawl jobs are run in parallel.