Indexed sources in AI Search
Summarize
Summary of Indexed sources in AI Search
Indexed sources in AI Search enable ServiceNow customers to designate specific ServiceNow AI Platform® tables or external document sets containing alphanumeric text and string fields for searchability. AI Search ingests and indexes this content to provide efficient search capabilities across platform records or external repositories. This functionality allows you to tailor which data is searchable, improving relevance and search performance.
Show less
Indexed Source Types
- Internal Indexed Source: Retrieves searchable text and metadata from ServiceNow AI Platform tables and their child tables, excluding certain platform tables that are not eligible for indexing. It cannot index remote tables.
- External Indexed Source: Retrieves searchable content from external repositories or remote tables by referencing an external content schema table rather than a native ServiceNow table.
Note: AI Search does not index Unicode characters in the High Surrogate Area; these are replaced with spaces.
Indexing and Search Performance
Search responsiveness is influenced by multiple factors you control:
- Index Size: Larger indexes take longer to search; avoid indexing unnecessary content.
- Number of Indexed Sources: More sources slow down search performance.
- Number of Indexed Fields: Increasing indexed fields extends search time independently of index size.
- Indexing Frequency: Frequent updates to indexed content increase resource competition, impacting response times.
Retention Policies and Filter Conditions
You can define retention policies and filter conditions to limit indexed records, reduce index size, and optimize search performance. For example, excluding records older than two years or those with certain statuses reduces data volume and indexing frequency. These settings also enable automatic purging of stale records.
Important: Retention policies are mandatory for indexed sources indexing records from the Task [task] table or its extensions, optional for others.
Attributes and Field Settings
Indexed sources can be customized at two levels:
- Attributes: Control indexing behavior for entire records from the source.
- Field Settings: Control indexing behavior for individual fields within records.
These settings allow fine-tuning of what content is included in the search index.
Indexing Behavior and Content Considerations
- AI Search automatically indexes changes (create, update, delete) in configured source tables and child tables.
- Unmodified records are indexed only after performing a full table index operation.
- Numeric fields are not directly searchable but can be used for filtering; to search numeric values, copy them into text/string fields.
- When indexing knowledge articles, content from knowledge blocks is included by default but can be excluded via configuration.
- Only one active indexed source per ServiceNow AI Platform table is allowed; duplicate sources created by plugins remain inactive unless manually enabled.
Additional Configuration Options
- Guardrails: Limit the number of task and alert records indexed to reduce index size and improve performance.
- Semantic Index Configuration: Use the AI Search generalized RAG framework to configure semantic indexing for records.
- Catalog Variable Indexing: Optionally enable indexing of catalog variable content on Catalog Item records and configure which items and variables are indexed.
Practical Benefits for ServiceNow Customers
By defining and configuring indexed sources appropriately, you can:
- Make relevant platform and external content searchable with AI Search.
- Optimize search speed and resource usage through filtering, retention policies, and selective field indexing.
- Maintain up-to-date search indexes reflecting record changes automatically.
- Customize indexing behavior to fit your organization’s data and search requirements.
Indexed sources designate ServiceNow AI Platform® tables and external document sets with alphanumeric text and string field content that you want to make searchable. AI Search ingests text and string fields from table records or external documents and stores their searchable alphanumeric content in its search index.
For instructions on creating an indexed source, see Create an indexed source.
Indexed source types
- Internal indexed source
- An internal indexed source retrieves alphanumeric content and metadata from text and string fields on ServiceNow AI Platform records. It includes a unique name and a reference to a ServiceNow AI Platform table with records that you want to make searchable. AI Search extracts and indexes searchable alphanumeric content and metadata from text and string fields on records in this table and in any of its child tables that you configure for indexing.
- External indexed source
- An external indexed source retrieves alphanumeric content and metadata from text and string fields of documents in an external repository or a remote table. It includes a unique name and a reference to an external content schema table instead of a ServiceNow AI Platform table. For more details on configuring indexed sources for external content, see Indexing and searching external content in AI Search.
Indexing and search performance
- Index size
- Indexing more content produces a larger index, which takes more time to search. Avoid indexing content that isn't needed for search.
- Number of indexed sources
- An index with more indexed sources takes longer to search than one with fewer indexed sources. This is true even if the two indexes are the same size.
- Number of indexed fields
- Increasing the number of fields you index across your indexed sources makes the system take longer to find search results. This effect is independent of index size and number of indexed sources.
- Indexing frequency
- The more often your indexed content is synchronized and updated, the more often search will compete with indexing for compute resources, increasing search response time. This is especially pertinent for indexed sources with frequently modified fields.
Indexed source retention policies and filter conditions
To limit the size of your index and the frequency of index updates, you can define retention policies and filter conditions for your indexed sources.
As an example, you can define a retention policy for an indexed source to exclude records that are more than two years old. This policy keeps your search results more current and reduces the size of your index. Changes made to the excluded records don't trigger index updates, so this policy also reduces indexing frequency.
Similarly, you can define a filter condition for an indexed source that excludes source table records with a specific status, such as Open. This filter condition reduces the number of records indexed from the source table, which in turn reduces the total amount of data you index. Excluding open records that have frequent updates also reduces indexing frequency.
AI Search also uses your retention policy and filter condition settings to automatically purge stale records from the index, reducing its size.
To learn more about creating retention policies and filter conditions for your indexed sources, see Indexed source retention policies and filter conditions.
Indexed source attributes and field settings
You can configure attributes and field settings for an indexed source to control indexing behavior for source records. Attributes control the indexed source's behavior at the record level, while field settings define its behavior for individual fields on indexed records. For more information, including lists of available attributes and field settings, see Indexed source attributes for AI Search and Field settings for AI Search.
Indexing content from an indexed source
Indexing content from knowledge articles
When indexing content from records in the Knowledge [kb_knowledge] table, AI Search defaults to including content defined in knowledge blocks. Administrators can override this default behavior and configure AI Search to exclude content from knowledge blocks when indexing knowledge articles. For details on making this change, see Exclude knowledge block content from the AI Search index.