Indexed source guardrails
Summarize
Summary of Indexed source guardrails
Indexed source guardrails in AI Search are designed to reduce index size and improve search performance by limiting the number of task and alert records indexed from large source tables like Task [task] and Alert [emalert]. These guardrails help prevent performance degradation caused by indexing excessively large data sets.
Show less
By default, AI Search indexes up to 10 million records from each of these tables and their child tables. Guardrails ensure only the most recently modified records are indexed, discarding older records when limits are exceeded. They apply after filter conditions and retention policies have limited the source records.
How Guardrails Work
- AI Search first checks the Guard Rail Limit for Indexed Data Sources [aisguardraillimitdatasource] table for configured limits per source.
- If no limit is found there, it checks the glide.ais.ingestion.guardrailsenableddatasources system property for limits.
- If neither source defines a limit, no guardrails apply to that indexed source.
- Guardrails limit indexed records after applying source filters and retention policies.
- Always indexes the most recent records, discarding older ones when the limit is exceeded.
Modifying Guardrail Settings
ServiceNow employees can adjust guardrail settings upon request:
- Override default record-count limits for Task and Alert tables
- Create custom guardrails for other source tables
- Disable guardrails if needed
Note that changes may take up to 24 hours to affect AI Search indexing behavior.
Impact on Indexing and Search Performance
Several factors controlled by customers influence search speed and indexing efficiency:
- Index size: Larger indexes take longer to search; avoid indexing unnecessary content.
- Number of indexed sources: More sources increase search time even if index size is constant.
- Number of indexed fields: More fields slow down search independently of index size or sources.
- Indexing frequency: Frequent updates can increase competition for resources, slowing search response times.
Retention Policies and Filter Conditions
To optimize index size and update frequency, customers can define retention policies and filters on indexed sources:
- Retention policies exclude older records (e.g., records older than 2 years), keeping search results current and reducing index size and update frequency.
- Filter conditions exclude records with certain attributes (e.g., open status) to reduce indexed record counts and indexing frequency.
- AI Search automatically purges stale records based on these policies.
- Retention policies are mandatory for indexed sources based on the Task table or its extensions; optional for others.
Reduce index size and increase search performance with guardrails that limit the number of task and alert source records indexed from indexed sources.
Guardrails overview
The Task [task] and Alert [em_alert] tables and their child tables contain large numbers of records. Indexing the full set of records from these tables for search increases the size of the AI Search index and can impact performance for indexing and search.
To reduce index size and preserve indexing and search performance, AI Search applies guardrails when indexing records from these tables. These guardrails limit the maximum number of records that can be indexed from the Task and Alert tables.
Guardrails are enabled in the base system for the Task and Alert tables and their child tables. By default, AI Search indexes a maximum of 10 million records for each of these tables.
How guardrails work
When guardrails are enabled, AI Search first checks the Guard Rail Limit for Indexed Data Sources [ais_guard_rail_limit_data_source] table to see whether a record exists for the indexed source (defining the maximum number of
records to index for that indexed source). If no table entry exists, AI Search checks the glide.ais.ingestion.guard_rails_enabled_datasources system property value to see whether a limit is defined there for the indexed source. If no limit is found in either place, AI Search does not apply guardrail limits to the indexed source.
Guardrail limits on the number of records indexed are applied after the set of source records is limited by the indexed source's filter conditions and retention policy. For details on indexed source filter conditions and retention policies, see Indexed source retention policies and filter conditions.
AI Search always indexes the most recently modified records from the indexed source table. If indexing causes the record count for the table to exceed the guardrail limit, AI Search discards older records from the index to make room for the newer records.
Modifying guardrail settings
- Override the base system record-count limits for the Task and Alert table guardrails
- Create custom guardrails to limit the maximum number of records indexed from other source tables
- Disable guardrails
Indexing and search performance
- Index size
- Indexing more content produces a larger index, which takes more time to search. Avoid indexing content that isn't needed for search.
- Number of indexed sources
- An index with more indexed sources takes longer to search than one with fewer indexed sources. This is true even if the two indexes are the same size.
- Number of indexed fields
- Increasing the number of fields you index across your indexed sources makes the system take longer to find search results. This effect is independent of index size and number of indexed sources.
- Indexing frequency
- The more often your indexed content is synchronized and updated, the more often search will compete with indexing for compute resources, increasing search response time. This is especially pertinent for indexed sources with frequently modified fields.
Indexed source retention policies and filter conditions
To limit the size of your index and the frequency of index updates, you can define retention policies and filter conditions for your indexed sources.
As an example, you can define a retention policy for an indexed source to exclude records that are more than two years old. This policy keeps your search results more current and reduces the size of your index. Changes made to the excluded records don't trigger index updates, so this policy also reduces indexing frequency.
Similarly, you can define a filter condition for an indexed source that excludes source table records with a specific status, such as Open. This filter condition reduces the number of records indexed from the source table, which in turn reduces the total amount of data you index. Excluding open records that have frequent updates also reduces indexing frequency.
AI Search also uses your retention policy and filter condition settings to automatically purge stale records from the index, reducing its size.
To learn more about creating retention policies and filter conditions for your indexed sources, see Indexed source retention policies and filter conditions.