Lemma and Unicode normalization
Summarize
Summary of Lemma and Unicode normalization
AI Search in ServiceNow automatically normalizes inflected words (lemmas) and Unicode glyphs during both indexing and search query processing. This normalization enhances search recall by allowing users to find content that contains variant forms of their search terms, such as different verb tenses, plurals, or accented characters. These normalization features are enabled by default and are not configurable.
Show less
Key Features
- Lemma normalization: AI Search identifies and normalizes the root forms (lemmas) of inflected terms across many languages, enabling matches between different word forms. For example, indexed terms like "selling" are expanded to include the lemma "sell," and queries for "sold" are similarly expanded, allowing results to match across these variants.
- Supported languages: Lemma normalization is language-specific and supports a wide range of languages including Arabic, Brazilian Portuguese, English, French, German, Japanese, Korean, Russian, Spanish, and others. Finnish uses algorithmic stemming to identify lemmas.
- Decompounding: For languages such as German, Danish, Hungarian, Korean, Norwegian (Bokmål), and Swedish, AI Search also indexes compound words along with their individual components, enhancing search accuracy for compound terms.
- Unicode normalization: AI Search normalizes Unicode characters to their nearest equivalents, enabling searches to match words regardless of accents or special glyphs. For example, "resumé" and "resume" are treated equivalently. This process uses Unicode normalization forms NFKD and NFKC.
- Interaction with other search features:
- Terms added by normalization do not trigger Genius Results configured with Term triggers.
- Normalized terms can trigger result improvement rules if matching query triggers.
- Stop words are removed before normalization and are not normalized.
- Synonyms are not normalized.
- Auto-corrected terms undergo lemma and Unicode normalization.
Benefits for ServiceNow Customers
This normalization capability allows ServiceNow users to perform more effective and flexible searches, finding relevant content even when the search query and indexed content use different word forms or Unicode variants. It reduces missed results due to linguistic variations and improves the overall user search experience without requiring manual configuration.
AI Search normalizes inflected words and Unicode glyphs during indexing and at search query time. Normalization improves search recall and enables users to find content with variant forms of their search query terms.
Normalization features are automatically enabled and aren't configurable.
Lemma normalization
Many languages include inflected forms of terms, such as plural nouns or verb tenses. AI Search normalizes inflected terms found in indexed content and search queries. Normalization enables matching based on a root form, such as the singular for a plural noun or the base form for a conjugated verb. This root form is called a lemma, and this process is referred to as lemma normalization.
For example, when a source record includes the conjugated verb selling, AI Search expands the indexed term to include the lemma form sell in addition to selling. When a user searches for the past-tense conjugated form sold, AI Search expands the search query term to include the lemma form sell as well as sold. Because the indexed term and the search query term include matching forms, the user's search returns the selling record as a result.
Decompounding
In addition to normalizing lemmas for German, Danish, Hungarian, Korean, Norwegian (Bokmål), and Swedish, AI Search indexes compound words and their individual component words. For example, when indexing a German record that contains the compound word Humanressourcen, AI Search indexes the component terms Human and ressourcen in addition to the compound term.
Unicode normalization
AI Search performs Unicode normalization on indexed terms and search query terms. This normalization makes alphabetical Unicode glyphs searchable using their nearest equivalent characters.
For example, when indexing a record containing the term resumé, AI Search expands the term to also include the non-accented form resume. This record appears as a search result when users search for either resume or resumé.
Unicode normalization includes NFKD (compatibility decomposition) and NFKC (compatibility composition) stages. For more information on these normalization forms, see the Unicode Standard Annex #15, https://www.unicode.org/reports/tr15/.
Interaction with other search features
The following table describes interactions between normalization and other search features.
| Feature | Interaction with lemma and Unicode normalization |
|---|---|
| Genius Results | Search query terms added by lemma or Unicode normalization can't trigger Genius Result configurations with Term trigger conditions. |
| Result improvement rules | A search query term added by lemma or Unicode normalization can trigger a result improvement rule if it matches the rule's Query trigger. |
| Stop words | If a search query term is defined as a stop word, AI Search removes that term without normalizing it. |
| Synonyms | If a search query term is defined as a synonym, AI Search doesn't normalize it. |
| Typo handling | AI Search performs lemma and Unicode normalization on auto-corrected search query terms. |