Search knowledge based using REST API efficiently

[wakespirit]

Dear all ,

I have some special case in Searching the knowledge base which are not working as expected using REST API.

The goal in exemple below is to return all KB where short description contains for instance "access VPN" or "VPN.

What I mean is that event if access keyword is missing I should be able to get "access vpn" result

For exemple :

https://myinstance.service-now.com/api/now/table/kb_knowledge?sysparm_query=short_descriptionLIKEaccess vpn 
– Fetches desired results (OK) as the whole string is provided

https://myinstance.service-now.com/api/now/table/kb_knowledge?sysparm_query=short_descriptionLIKEvpn 
– Not all results are return as access is missing (MISSING ARTICLES)

The only way we make it works is to specify the whole querry as below:

https://myinstance.service-now.com/api/now/table/kb_knowledge?sysparm_query=short_descriptionLIKEacc... 
– Fetches desired results

 1^st Query fetches the desired results only if all words Access and VPN are in sequence. If you change the sequence, no results are returned.

2^nd Query does not fetch all the results. As word access is missing, it won’t return the results with Access and VPN in short_description

 3^rd Query fetches all desired results; However as you can see I’ll have to append all the words with prefix short_descriptionLIKE. This will be a too long query if I ma more criterai.

For example, consider someone searching for kb with ‘I want to know how to get access to my application on Mobile using F5’.

Then by this long sentence I should be able to retrive any Kb wich has suitable matching short description. Which means that short descritpion can be only "access application on mobile".

Any idea how to handle such sear result ?

regards

[asifnoor]

Hi Wakespirit,

Kindly mark the comment(s) as helpful if this has helped you in solving the problem.

[moers]

A 'contains' or 'like' of each word in the search term will not help.

E.g if you have a "I need help with my laptop" query, what you actually want is a query like  (description contains "help" AND description contains "laptop"). Otherwise you will only find records where all words in the term do match.

 To achieve this, the stop words must be excluded from the list. This can be done server side (requires a custom REST API) or on client (requires the stop list to be available)

• get a list of stop words an e.g. save it in a sys_prop
  https://www.link-assistant.com/seo-stop-words.html
  https://www.ranks.nl/stopwords

• split the query and remove all words found in the stop list
  

  var query = 'I need help with my laptop'; // get the query from the http request
  
  var stopWords = gs.getProperty('stopword.list').toString().split(/[,\s]+/);
  var querySplit = query.toLowerCase().match(/\S+/g);
  var arrayUtil = new ArrayUtil();
  var queryArray = arrayUtil.diff(querySplit, stopWords);
  
  var query = queryArray.reduce(function (out, segment) {
    if (segment.length > 0) {
      out.push('short_descriptionLIKE' + segment);
    }
    return out;
  }, []);
  
  var kb = new GlideRecord('kb_knowledge');
  kb.addEncodedQuery(query.join('^'));
  kb.addQuery('active', 'true');​
  // ... 

[wakespirit]

Moers, thnaks for your help

I have try your suggestion. I have place in a sys prop the word list as a test for stop words based on your sample sentence

 stop.list = I,need,with,my

When I check my log at different point, your queery build is taking in a account the first character "I" which is render in querry as "i" which is part of stop words

As this character is part of stop word how can it be in query ?

Question : Does the querry will work if we use also LAPTOP and HELP in reverse order ?

What this line of code does exactly ?

kb.addEncodedQuery(query.join('^'));

Here is my complete script I build with a sentence which should return something

gs.info('REST API REQUEST')	;
	var query = 'How to add signature to email'; // get the query from the http request

var stopWords = gs.getProperty('stopword.list').toString().split(/[,\s]+/);
gs.info('REST API STOP WORD:' + stopWords)	;

var querySplit = query.toLowerCase().match(/\S+/g);
gs.info('REST API QUERY LOWERCASE :' + querySplit)	;
var arrayUtil = new ArrayUtil();
var queryArray = arrayUtil.diff(querySplit, stopWords);
gs.info('REST API DIFF WORD:' + queryArray)	;
	
query = queryArray.reduce(function (out, segment) {
  if (segment.length > 0) {
    out.push('short_descriptionLIKE' + segment);
  }
  return out;
}, []);

gs.info('REST API :' + query)	;
	
var kb = new GlideRecord('kb_knowledge');
kb.addEncodedQuery(query.join('^'));
kb.addQuery('active', 'true');
kb.query();
	gs.info('REST KB RECORD COUNT :' + kb.getRowCount());
	if (kb.next())
		{
			response.setBody(kb);
		}

 Is itb the correct way to return the response with the KB glide record because I get a an error of Unexpected string in json when testing it.

Thanks for your feedback

regards

[moers · Accepted solution]

Hi wakespirit

the stop words list must have the same case as the query.

var stopWords = gs.getProperty('stopword.list').toString().toLowerCase().split(/[,\s]+/);

check if the stopWords an array? Format must be "term a, term b, etc";

gs.info(Array.isArray(stopWords))

same with queryArray

var arrayUtil = new ArrayUtil();

var queryArray = arrayUtil.diff(querySplit, stopWords); (querySplit - stopWords)

gs.info(Array.isArray(queryArray) + ' : '  + JSON.stringify(queryArray))

if query is an array, query.join('^') results in short_descriptionLIKEsegment_1^short_descriptionLIKEsegment_2 (desc contains [0] AND desc contains [1] AND....)

you can print the query with gs.info(kb.getEncodedQuery())

response.setBody(kb); might not work as kb is a GlideRecord object, create a json object e.g. 

response.setBody({ id : kb.getValue('sys_id'), desc : kb.getValue('short_description)});

Cheers

Bori