Indexed sources in AI Search
Summarize
Summary of Indexed sources in AI Search
Indexed sources in AI Search allow ServiceNow customers to make alphanumeric text and string content from ServiceNow AI Platform® tables or external document repositories searchable. AI Search ingests this content and stores it in a search index to enable efficient retrieval. Indexed sources can be either internal (ServiceNow platform tables) or external (content from external repositories or remote tables).
Show less
Indexed Source Types
- Internal indexed sources: Retrieve and index text and string fields from ServiceNow AI Platform tables and their child tables. Some platform tables are excluded from indexing. Remote tables cannot be indexed internally.
- External indexed sources: Ingest content from external repositories or remote tables via external content schema tables.
Indexing and Search Performance Considerations
Several factors impact AI Search performance and must be managed carefully:
- Index size: Larger indexes take longer to search. Avoid indexing unnecessary content and large tables like Task [task].
- Number of indexed sources: More sources increase search time, even if index size is similar.
- Number of indexed fields: Indexing more fields slows search independently of index size or source count.
- Indexing frequency: Frequent updates can compete with search resources and degrade response times, especially for frequently modified records.
Retention Policies and Filter Conditions
To optimize index size and update frequency, customers can define retention policies and filter conditions on indexed sources. For example, excluding records older than two years or filtering out frequently updated statuses like "Open" reduces indexing load and keeps search results current. AI Search also automatically purges stale records based on these settings. Retention policies are mandatory for Task [task] tables and their extensions.
Indexed Source Configuration
- Attributes: Control indexing behavior for entire records from a source.
- Field settings: Customize indexing behavior at the individual field level.
These configurations enable tailored indexing to suit specific search requirements.
Indexing Behavior and Content Details
- AI Search automatically indexes create, update, and delete operations on records in indexed sources and their child tables.
- Unmodified records require a full table index operation to be included.
- Only text and string fields are indexed for search; numeric fields can be used for filtering but not for search queries unless copied into string fields.
- When indexing knowledge articles, AI Search includes knowledge block content by default but can be configured to exclude it.
- Only one indexed source can be active per ServiceNow AI Platform table, preventing duplicate active sources.
Additional Features and Best Practices
- Guardrails: Limit indexing of task and alert records to reduce index size and improve performance.
- Semantic index configuration: Use AI Search's generalized RAG framework for enhanced semantic indexing of records.
- Catalog variable indexing: Enable indexing of variables on Catalog Item records with configurable scope.
Practical Guidance for ServiceNow Customers
By defining appropriate indexed sources, configuring retention policies, and carefully selecting fields and tables to index, ServiceNow customers can optimize AI Search performance and relevance. This enables efficient, up-to-date search results across internal and external content sources while managing resource utilization and index size.
Indexed sources designate ServiceNow AI Platform® tables and external document sets with alphanumeric text and string field content that you want to make searchable. AI Search ingests text and string fields from table records or external documents and stores their searchable alphanumeric content in its search index.
For instructions on creating an indexed source, see Create an indexed source.
Indexed source types
- Internal indexed source
- An internal indexed source retrieves alphanumeric content and metadata from text and string fields on ServiceNow AI Platform records. It includes a unique name and a reference to a ServiceNow AI Platform table with records that you want to make searchable. AI Search extracts and indexes searchable alphanumeric content and metadata from text and string fields on records in this table and in any of its child tables that you configure for indexing.
- External indexed source
- An external indexed source retrieves alphanumeric content and metadata from text and string fields of documents in an external repository or a remote table. It includes a unique name and a reference to an external content schema table instead of a ServiceNow AI Platform table. For more details on configuring indexed sources for external content, see Indexing and searching external content in AI Search.
Indexing and search performance
- Index size
- Indexing more content produces a larger index, which takes more time to search. Avoid indexing content that isn't needed for search.
- Number of indexed sources
- An index with more indexed sources takes longer to search than one with fewer indexed sources. This is true even if the two indexes are the same size.
- Number of indexed fields
- Increasing the number of fields you index across your indexed sources makes the system take longer to find search results. This effect is independent of index size and number of indexed sources.
- Indexing frequency
- The more often your indexed content is synchronized and updated, the more often search will compete with indexing for compute resources, increasing search response time. This is especially pertinent for indexed sources with frequently modified fields.
Indexed source retention policies and filter conditions
To limit the size of your index and the frequency of index updates, you can define retention policies and filter conditions for your indexed sources.
As an example, you can define a retention policy for an indexed source to exclude records that are more than two years old. This policy keeps your search results more current and reduces the size of your index. Changes made to the excluded records don't trigger index updates, so this policy also reduces indexing frequency.
Similarly, you can define a filter condition for an indexed source that excludes source table records with a specific status, such as Open. This filter condition reduces the number of records indexed from the source table, which in turn reduces the total amount of data you index. Excluding open records that have frequent updates also reduces indexing frequency.
AI Search also uses your retention policy and filter condition settings to automatically purge stale records from the index, reducing its size.
To learn more about creating retention policies and filter conditions for your indexed sources, see Indexed source retention policies and filter conditions.
Indexed source attributes and field settings
You can configure attributes and field settings for an indexed source to control indexing behavior for source records. Attributes control the indexed source's behavior at the record level, while field settings define its behavior for individual fields on indexed records. For more information, including lists of available attributes and field settings, see Indexed source attributes for AI Search and Field settings for AI Search.
Indexing content from an indexed source
Indexing content from knowledge articles
When indexing content from records in the Knowledge [kb_knowledge] table, AI Search defaults to including content defined in knowledge blocks. Administrators can override this default behavior and configure AI Search to exclude content from knowledge blocks when indexing knowledge articles. For details on making this change, see Exclude knowledge block content from the AI Search index.