Indexed source guardrails
Summarize
Summary of Indexed source guardrails
Indexed source guardrails in ServiceNow’s AI Search manage the indexing of records from large tables like Task [task] and Alert [emalert] to optimize search performance and control index size. By limiting the number of task and alert source records indexed, these guardrails prevent performance degradation caused by excessively large indexes.
Show less
How Guardrails Work
Guardrails limit the maximum number of records indexed from specified tables. By default, AI Search indexes up to 10 million records for each of the Task and Alert tables and their child tables. The system checks specific tables and system properties to determine if a guardrail limit applies to an indexed source. If the record count exceeds the limit, older records are discarded to prioritize indexing of the most recent data. Guardrails apply after filter conditions and retention policies are enforced on the source records.
Modifying Guardrail Settings
ServiceNow personnel can adjust guardrail settings by:
- Overriding the default record-count limits for Task and Alert tables
- Creating custom guardrails for other source tables
- Disabling guardrails if needed
Note that changes to guardrail settings may take up to 24 hours to reflect in AI Search indexing behavior.
Indexing and Search Performance Considerations
Several factors controlled by customers influence AI Search performance:
- Index size: Larger indexes take longer to search; avoid indexing unnecessary content.
- Number of indexed sources: More sources increase search time even if the index size is the same.
- Number of indexed fields: More fields indexed increases search response time independently of index size.
- Indexing frequency: Frequent updates to indexed content can cause search and indexing to compete for resources, slowing response time.
Retention Policies and Filter Conditions
To control index size and update frequency, customers should define retention policies and filter conditions on indexed sources. These help exclude outdated or unnecessary records, such as excluding records older than two years or those with specific statuses like Open. This not only reduces index size but also decreases indexing frequency by preventing updates from excluded records from triggering reindexing.
Retention policies are mandatory for indexed sources involving the Task table or its extensions and optional for other sources. AI Search also uses these settings to automatically purge stale records from the index, maintaining optimal size and performance.
Reduce index size and increase search performance with guardrails that limit the number of task and alert source records indexed from indexed sources.
Guardrails overview
The Task [task] and Alert [em_alert] tables and their child tables contain large numbers of records. Indexing the full set of records from these tables for search increases the size of the AI Search index and can impact performance for indexing and search.
To reduce index size and preserve indexing and search performance, AI Search applies guardrails when indexing records from these tables. These guardrails limit the maximum number of records that can be indexed from the Task and Alert tables.
Guardrails are enabled in the base system for the Task and Alert tables and their child tables. By default, AI Search indexes a maximum of 10 million records for each of these tables.
How guardrails work
When guardrails are enabled, AI Search first checks the Guard Rail Limit for Indexed Data Sources [ais_guard_rail_limit_data_source] table to see whether a record exists for the indexed source (defining the maximum number of
records to index for that indexed source). If no table entry exists, AI Search checks the glide.ais.ingestion.guard_rails_enabled_datasources system property value to see whether a limit is defined there for the indexed source. If no limit is found in either place, AI Search does not apply guardrail limits to the indexed source.
Guardrail limits on the number of records indexed are applied after the set of source records is limited by the indexed source's filter conditions and retention policy. For details on indexed source filter conditions and retention policies, see Indexed source retention policies and filter conditions.
AI Search always indexes the most recently modified records from the indexed source table. If indexing causes the record count for the table to exceed the guardrail limit, AI Search discards older records from the index to make room for the newer records.
Modifying guardrail settings
- Override the base system record-count limits for the Task and Alert table guardrails
- Create custom guardrails to limit the maximum number of records indexed from other source tables
- Disable guardrails
Indexing and search performance
- Index size
- Indexing more content produces a larger index, which takes more time to search. Avoid indexing content that isn't needed for search.
- Number of indexed sources
- An index with more indexed sources takes longer to search than one with fewer indexed sources. This is true even if the two indexes are the same size.
- Number of indexed fields
- Increasing the number of fields you index across your indexed sources makes the system take longer to find search results. This effect is independent of index size and number of indexed sources.
- Indexing frequency
- The more often your indexed content is synchronized and updated, the more often search will compete with indexing for compute resources, increasing search response time. This is especially pertinent for indexed sources with frequently modified fields.
Indexed source retention policies and filter conditions
To limit the size of your index and the frequency of index updates, you can define retention policies and filter conditions for your indexed sources.
As an example, you can define a retention policy for an indexed source to exclude records that are more than two years old. This policy keeps your search results more current and reduces the size of your index. Changes made to the excluded records don't trigger index updates, so this policy also reduces indexing frequency.
Similarly, you can define a filter condition for an indexed source that excludes source table records with a specific status, such as Open. This filter condition reduces the number of records indexed from the source table, which in turn reduces the total amount of data you index. Excluding open records that have frequent updates also reduces indexing frequency.
AI Search also uses your retention policy and filter condition settings to automatically purge stale records from the index, reducing its size.
To learn more about creating retention policies and filter conditions for your indexed sources, see Indexed source retention policies and filter conditions.