Hi All
Recently, I received a request from a Japanese customer to improve AI search for Korean. As you know, AI Search does not support Korean. Indexing Korean text is treated as English (space delimited) because the Now Platform does not have Korean tokenization capabilities.
When you search for words in your search input, the AI search query finds results where the keywords in the source index match the words separated by space. However, Korean does not always follow space word boundary rules for user content and search keywords. Therefore, search terms do not return exact results. I believe this issue occurs with all other languages for which the ServiceNow platform does not support language tokenization.
We all know that meta fields help improve search results. But manually reviewing all the content and creating the meta is a lot of work. Here is a workaround to update the metadata through scripts.
However, you may be wondering what keywords to use for meta fields and where to find them. In fact, the platform has a table where you can find the keywords that users searched for and analyze the results. This table name is "sys_search_event". Let's open the table records in the list.
Here, you can see that the value = false in the "Has Results" column means that no result was found. However, in order to be able to handle future content without worrying about this "Has Results" value, we will check whether all keywords are included in the knowledge content and set them to be registered in the meta field. Then, the next time a user searches, their search will be found in the meta so they can find the article they're looking for.
For results improvement, there is no point in registering stop words in meta, so we can create a stop word dictionary and check for search keywords that are not listed in the dictionary.
Developers can monitor the search keywords list regularly and run scheduled jobs when needed to improve search results.
// Get Korean stop word dictionary
var stopWordArray = [];
var grDic = new GlideRecord('ais_dictionary_term');
grDic.addQuery('dictionary=9a34883187fcbd10d91f7738dabb3576');
grDic.query();
while (grDic.next()) {
var stopWord = grDic.getValue('term');
stopWordArray.push(stopWord);
}
var grSE = new GlideRecord('sys_search_event');
//grSE.addQuery('has_results', 'false');
grSE.addQuery('language', 'ko');
grSE.query();
//Adding all has no result text into array
var srchArray = [];
while (grSE.next()) {
var keyWord = grSE.getValue('search_query');
// Check value isNotEmpty or Null
// Check value is not exist in the list(Distinct list)
// Ignore stop word meta registration
if (keyWord && srchArray.indexOf(keyWord) == -1 && stopWordArray.indexOf(keyWord) == -1) {
srchArray.push(keyWord);
}
}
//If the search keyword is included in the body of the knowledge article, add it to the meta item
for (var i = 0; i < srchArray.length; i++) {
var srTxt = srchArray[i];
var gr = new GlideRecord('kb_knowledge');
gr.addQuery('workflow_state', 'published');
gr.addQuery('language', 'ko');
gr.addQuery('text', 'CONTAINS', srTxt);
gr.query();
//Iterate and output results
while (gr.next()) {
var metaVal = gr.getValue('meta');
if (metaVal) {
var myArray = metaVal.split(",");
//Add the search keyword to the meta item only if it does not exist
if (myArray.indexOf(srTxt) == -1) {
myArray.push(srTxt);
gr.setValue("meta", myArray.toString());
gr.setWorkflow(false);
gr.update();
}
} else {
//If the meta field is blank, set the value without commas.
gr.setValue("meta", srTxt);
gr.setWorkflow(false);
gr.update();
}
}
} //End For loop
gs.log('Update Knowledge table meta value for Korean completed');
Let's test the keyword using "외국인" and search in the Service Portal. I can see No results found message as shown on the screen below.
Now I am running the script, and the knowledge article meta fields are updated as shown in the screenshot below.
Now again I search the keyword using "외국인" in Service Portal. I can see the results found as shown on the screen below.
