Indexed sources in AI Search
Summarize
Summary of Indexed sources in AI Search
Indexed sources in AI Search allow ServiceNow customers to designate specific ServiceNow AI Platform® tables or external document sets containing alphanumeric text and string fields to be searchable. AI Search ingests and indexes this content to enable efficient search across selected records and documents.
Show less
Indexed Source Types
- Internal Indexed Source: Retrieves and indexes text and string fields from records in a specified ServiceNow AI Platform table and its child tables. Some tables are excluded from indexing and remote tables cannot be indexed internally.
- External Indexed Source: Retrieves and indexes text and string fields from documents stored in an external repository or remote table by referencing an external content schema table.
Note: AI Search does not index certain Unicode characters (High Surrogate Area), replacing them with spaces.
Indexing and Search Performance Considerations
- Index Size: Larger indexes take longer to search, so avoid indexing unnecessary content, especially large tables like Task [task].
- Number of Indexed Sources: More sources in an index increase search time, even if total size is unchanged.
- Number of Indexed Fields: Increasing indexed fields slows search independently of index size.
- Indexing Frequency: Frequent updates compete with search resources, affecting response time, especially for frequently modified data.
Retention Policies and Filter Conditions
To manage index size and update frequency, customers can configure retention policies (e.g., exclude records older than two years) and filter conditions (e.g., exclude records by status). These settings also help automatically purge stale records, improving search efficiency.
Important: Retention policies are required for indexed sources involving the Task [task] table or its extensions, and optional for others.
Indexed Source Configuration
- Attributes: Control indexing behavior at the record level.
- Field Settings: Define indexing behavior for individual fields on indexed records.
These configurations allow customization of how records and fields are indexed, optimizing search relevance and performance.
Indexing Behavior and Limitations
- Indexing begins automatically after defining an indexed source to reflect create, update, and delete operations.
- Unmodified records are indexed only after a full table index is performed.
- Numeric fields can be used for faceting or filtering search results but are not searchable unless their values are copied into text or string fields.
- When indexing knowledge articles, AI Search includes content in knowledge blocks by default; administrators can configure this to exclude knowledge block content if desired.
- Only one active indexed source per ServiceNow AI Platform table is allowed; duplicates are made inactive by default to prevent conflicts.
Additional Features and Best Practices
- Guardrails: Limit the number of task and alert records indexed to reduce index size and improve search performance.
- Semantic Index Configuration: Utilize the generalized RAG framework to configure semantic indexing for records from ServiceNow tables.
- Catalog Variable Indexing: Activate and configure indexing of searchable content from variables on Catalog Item records.
Practical Benefits for ServiceNow Customers
By defining and configuring indexed sources effectively, customers can tailor AI Search to their organizational needs, ensuring relevant, timely, and performant search results across internal and external content. Proper use of retention policies, filter conditions, and indexing configurations helps maintain manageable index sizes and optimal search response times.
Indexed sources designate ServiceNow AI Platform® tables and external document sets with alphanumeric text and string field content that you want to make searchable. AI Search ingests text and string fields from table records or external documents and stores their searchable alphanumeric content in its search index.
For instructions on creating an indexed source, see Create an indexed source.
Indexed source types
- Internal indexed source
- An internal indexed source retrieves alphanumeric content and metadata from text and string fields on ServiceNow AI Platform records. It includes a unique name and a reference to a ServiceNow AI Platform table with records that you want to make searchable. AI Search extracts and indexes searchable alphanumeric content and metadata from text and string fields on records in this table and in any of its child tables that you configure for indexing.
- External indexed source
- An external indexed source retrieves alphanumeric content and metadata from text and string fields of documents in an external repository or a remote table. It includes a unique name and a reference to an external content schema table instead of a ServiceNow AI Platform table. For more details on configuring indexed sources for external content, see Indexing and searching external content in AI Search.
Indexing and search performance
- Index size
- Indexing more content produces a larger index, which takes more time to search. Avoid indexing content that isn't needed for search.
- Number of indexed sources
- An index with more indexed sources takes longer to search than one with fewer indexed sources. This is true even if the two indexes are the same size.
- Number of indexed fields
- Increasing the number of fields you index across your indexed sources makes the system take longer to find search results. This effect is independent of index size and number of indexed sources.
- Indexing frequency
- The more often your indexed content is synchronized and updated, the more often search will compete with indexing for compute resources, increasing search response time. This is especially pertinent for indexed sources with frequently modified fields.
Indexed source retention policies and filter conditions
To limit the size of your index and the frequency of index updates, you can define retention policies and filter conditions for your indexed sources.
As an example, you can define a retention policy for an indexed source to exclude records that are more than two years old. This policy keeps your search results more current and reduces the size of your index. Changes made to the excluded records don't trigger index updates, so this policy also reduces indexing frequency.
Similarly, you can define a filter condition for an indexed source that excludes source table records with a specific status, such as Open. This filter condition reduces the number of records indexed from the source table, which in turn reduces the total amount of data you index. Excluding open records that have frequent updates also reduces indexing frequency.
AI Search also uses your retention policy and filter condition settings to automatically purge stale records from the index, reducing its size.
To learn more about creating retention policies and filter conditions for your indexed sources, see Indexed source retention policies and filter conditions.
Indexed source attributes and field settings
You can configure attributes and field settings for an indexed source to control indexing behavior for source records. Attributes control the indexed source's behavior at the record level, while field settings define its behavior for individual fields on indexed records. For more information, including lists of available attributes and field settings, see Indexed source attributes for AI Search and Field settings for AI Search.
Indexing content from an indexed source
Indexing content from knowledge articles
When indexing content from records in the Knowledge [kb_knowledge] table, AI Search defaults to including content defined in knowledge blocks. Administrators can override this default behavior and configure AI Search to exclude content from knowledge blocks when indexing knowledge articles. For details on making this change, see Exclude knowledge block content from the AI Search index.