Configuring SharePoint Online Search Connector

[SmritiGupta]

Note: As part of the Yokohama release (EA on 30th Jan), we are deprecating the SharePoint Online Search Connector and replacing it with a new, Raytion-based External Content Connector (version 1.0.9) application. Refer to the External Search Connector article for details around license and plugin requirements.

 

Refer to External Content Connectors product documentation for additional details.

 

The ServiceNow SharePoint Online Search Connector offers a modern consumer-grade search configuration to manage your information resources.

As search is the key to productivity, SharePoint Online Search Connector (SPOC) helps your employees to search and discover content from the ServiceNow employee experience. You can index content from multiple sites, enabling employees to find the relevant information through Employee Center, Service Portal, Now Mobile app, and Virtual Agent.

Here's a step by step Instruction guide organized across two parts for configuring SharePoint Online Search Connector on your instance. The first part will cover configuring Azure configurations and the second part will cover configuring ServiceNow configurations. 

Part 1: Azure Configuration

Recording 

Video to access the steps - Azure Setup

1. Access Azure Portal:

    
   • Log in to the Azure Portal. 

2. Create or Select App:

   • Navigate to "Azure services" and select "App Registrations." 
   • Create a new app registration or select an existing one. 

3. Obtain Client ID and Tenant ID:

    
   • Once inside the app, note down the "Client ID" (Application ID). 
   • Remember that the "Tenant ID" is common for the SharePoint instance. 

4. Create Client Secret:

    
   • In the app settings, go to "Certificates & secrets." 
   • Create a new client secret with a description and expiration. 
   • Store the generated client secret securely.[Important] 

5. Upload Certificate:

    
   • Move to the "Certificates & secrets" tab. 
   • Upload the necessary certificate file (e.g., .cer file or .certificate). Follow the below steps to generate a certificate file (.jks and .cer file pairs) 
   • Note the "Thumbprint" of the uploaded certificate.[Store the thumbprint to be added later in the ServiceNow instance] 

If you are looking for ways to generate .jks and .cer file, here are some helpful commands to generate .jks and .cer file using java keytool 

keytool -genkey -alias <filename> -keyalg RSA -keystore <filename>.jks -storepass <password> -validity 360 -keysize 2048  

keytool -export -alias <filename> -file <filename>.cer -keystore <filename>.jks -storepass <password>

You would need the jks file while later in ServiceNow configurations 

6. Add API Permissions:

    
   • Navigate to "API permissions" within the app settings. 

7. Add Microsoft Graph Permissions:

    
   • Click "Add a permission" > "Microsoft Graph." 
   • Choose "Delegated permissions"  
   • Go to sites and add "Sites.Read.All" permission. 
   • Go to the user module and select User.read 

8. Add Rest Permissions:

    
   • Click "Add permissions" > "SharePoint." 
   • Select "Application permissions"  
   • Give "Sites.FullControl.All" and "User.Read.All" permissions.[TO BE CHANGED] 

9. Grant Admin Consent:

    
   • Grant admin consent for the added permissions. 

10. Configure Redirect URL:

     
    • Move to the "Authentication" tab. 
    • In the "Redirect URIs" section, add your ServiceNow instance URL followed by "/oauth_redirect.do" or the appropriate path. Eg- https:///oauth_redirect.do 

11. Complete Configuration:

     
    • Save your changes and ensure all required configurations are completed on the Azure portal side. 

Part 2: ServiceNow Configuration 

Step 1 

 

Recording 

Video to access the steps - ServiceNow Configuration Step 1 

Prerequisites 

1. Obtain the Client ID, Client Secret, and Tenant ID from the Azure application created for the connector. 
2. Generate a Java Key Store file and a certificate to be used for authentication. 
3. Know the SharePoint host URL. 

Create Connections 

1. Go to "Application Registry" in ServiceNow. 
2. For more details go to "Setting up SharePoint Online" section in product documentation 
3. Create a record for the SharePoint Rest connection: 
   • Name: Choose a name (e.g., SharePoint Rest). 
   • Client ID: Use the Azure Client ID. 
   • Client Secret: Use the Azure Client Secret. 
   • Script: Refer to script - OauthUtilSPJWTOnline 
   • Grant Type: Client Credential. 
   • Token URL: Replace Tenant ID with the tenant id captured from the azure application and add it to the url -- https://login.microsoftonline.com/<TentantID>/oauth2/token 
   • Redirect URL: Use the SharePoint instance URL followed by oauth_redirect.do 
   • Save the record. 

4. Create another record for the SharePoint Graph connection: 
   • Name: Choose a name (e.g., SharePoint Graph). 
   • Client ID: Use the Azure Client ID. 
   • Client Secret: Use the Azure Client Secret. 
   • No need for authorization script 
   • Grant Type: Authorization Code. 
   • Authorization URL: Replace placeholders with Tenant ID. - https://login.microsoftonline.com/ <TentantID>/oauth2/v2.0/authorize 
   • Token URL: Replace Tenant ID with the tenant id captured from the azure application and add it to the url -- https://login.microsoftonline.com/<TenantID>/oauth2/token 
   • Token Revocation URL: Use the SharePoint instance URL followed by oauth_redirect.do 
   • Save the record. 

Verify OAuth Entity Profile Table 

1. Verify the existence of default profiles for Graph and Rest connections. 

OAuth Entity Scopes Table 

Create entries for Graph and Rest: 

1. For Graph: 
   • Name: Choose a name (e.g., Graph Scope). 
   • Provider: Use the Graph application registry record which we created in first step 
   • Scope: Use the provided scope in documentation - offline_access https://graph.microsoft.com/Sites.ReadWrite.All  

2. For Rest: 
   • Name: Choose a name (e.g., Rest Scope). 
   • Provider: Use the Rest application registry record. 
   • Scope: Use the ServiceNow instance URL followed by default. eg - https://.sharepoint.com/.default 

OAuth Entity Profile Scope Table 

Create entries for Graph and Rest: 

1. For Graph: 
   • Profile: Choose the default Graph profile as referred to in the Step - Verify OAuth Entity Profile Table 
   • Scope: Choose the Graph scope created in the OAuth Entity Scopes configuration 

2. For Rest: 
   • Profile: Choose the default rest profile as referred to in the Step - Verify OAuth Entity Profile Table 
   • Scope: Choose the Rest scope created in the OAuth Entity Scopes configuration 

Step 2 

Recording 

Video to access the steps - ServiceNow Configuration Step 2  

OAuth 2.0 Credentials 

1. Go to "OAuth 2.0 Credentials" in ServiceNow. 
2. Create a record for the Graph connection: 
   • Name: Choose a name. 
   • Profile: Choose the Graph profile. 

3. Create another record for the Rest connection: 
   • Name: Choose a name. 
   • Profile: Choose the Rest profile. 

Create Connections

1. Go to the "Connections" table. 
2. Create an HTTPS connection for Graph: 
   • Name: Choose a name. 
   • Credential: Select the Graph credential. 
   • Connection Alias: Use the provided alias - sn_sp_spoke.MicrosoftSharePointGraph 
   • Connection URL: Use the provided URL. - https://graph.microsoft.come/v1.0/ 

3. Create another HTTPS connection for Rest: 
   • Name: Choose a name. 
   • Credential: Select the Rest credential. 
   • Connection Alias: Use the provided alias. - sn_sp_spoke.MicrosoftSharepointOnline 
   • Connection URL: Use the ServiceNow instance host URL. - https:/// 

Create Tenants 

1. Go to the "Tenants" table. 
2. Create a tenant entry for Graph: 
   • Alias: Use the provided alias - sn_sp_spoke.MicrosoftSharePointGraph 
   • Tenant ID and Resource Id is auto populated 
   • SharePoint Root URL: Use the provided SharePoint root URL. 
   • Instance – Choose instance as online  

3. Create another tenant entry for Rest: 
   • Alias: Use the provided alias. - sn_sp_spoke.MicrosoftSharePointOnline 
   • Tenant ID: Use the Rest tenant ID. 
   • SharePoint Root URL: Use the provided SharePoint root URL. 

Certificates and JWT Keys 

1. Go to "Certificate" in ServiceNow. Navigate to System Definition -> Certificates -> Microsoft SharePoint Online Certificate 
2. Open the Microsoft SharePoint Online certificate uploaded earlier in section "Upload Certificate" above 
3. Attach the JKS file and enter its password.  
4. Save the record.

    

Note: 

If glide.security.file.mime_type.validation - the value is set to true, then it won't allow the jks file to upload. 

Will throw an error File type not permitted or mime type does not match the file content. 

Either turn the value to false or add jks file extension in glide.attachment.extensions  

JWT Keys 

1. Go to "JWT Keys" and open the JWT key. 
2. Enter the JKS file password and save. 

JWT Providers

1. Go to "JWT Providers" and open the provided JWT provider record available OOB. 
2. Update the "Aud" claim with the url provided in documentation.  
3. Update the "iss" and "sub" claims with the Client ID. 

Connection Thumbprint

1. Open the Rest connection. 
2. Go to "Attributes" and enter the Thumbprint value extracted earlier. 
3. Save the record. 

Get Tokens 

1. Go to OAuth2.0 Credentials section and use the "Get Authorization Token" related link for the Rest connection to fetch the Rest token. 
2. Similarly, use the related link for the Graph connection to fetch the Graph token. 

Step 3 

Recording 

Video recording can be found here - ServiceNow Configurations Step 3 

EVAM Configuration 

User Credential Configuration 

1. Go to the "Connection and Credentials" table. 
2. Open the "SharePoint Ingest User" record (an out-of-box record). 
3. Create a new basic auth credential record: 
   • Name: Choose a name (e.g., SharePoint). 
   • Username: Specify the SharePoint user's username. 
   • Password: Enter the user's password. 
   • Ensure the user has the "ais_external_content" role assigned. 

4. Submit the record. 

Confirm AIS Search Server is configured 

1. Ensure an AIS server is connected and operational. 

EVAM Definitions and Bundles

1. Go to the "EVAM Definitions" table. 
2. Open the "ESC Portal Search" EVAM definition. 
3. Navigate to the "M2M Bundle" tab. 
4. Link an existing bundle, and select "SharePoint Search Bundle."[Available OOB] 
5. Specify an order and submit the record. 
6. Repeat the same process for "Virtual Agent Search," ensuring the proper bundle is linked. 

Search Profile Configuration

1. Go to the "Search Profiles" table. 
2. Open the "ESC Portal Default Search Profile" record. 
3. In the "Search Sources" tab, link the "SharePoint" search Source. 
4. In the same record, ensure the following: 
   • Set the state to "Published" for stop words and synonyms. 
5. After making changes, publish the record. 

Step 4 

Recording 

Recording can be found here - ServiceNow Configurations Step 4  

1. Configure Now Mobile Default Search Profile

   • Perform the same steps for the Now Mobile Default Search Profile as you did for the ESC Portal Default Search Profile. 
   • Follow the instructions provided earlier to link the SharePoint Search Source and configure any other necessary settings. 

2. Update Virtual Agent Default Search Engine

   • Go to the "Virtual Agent Default Search Engine" table. 

   • Open the record and update it with the following values:  

3. Add SharePoint Connected Script 

   • In the same record, go to the specified tab. 
   • Add a new script for SharePoint integration.   

Global Search M2M Item Configuration 

Navigate to Global Search M2M Item Configuration (sys_sg_global_search_m2m_master_item) and create the following:

Global Search Configuration: Homepage Search - Catalog, Knowledge, People 
Item Configuration: Sharepoint External Search Excel Main Item 
Application: SharePoint Search Connector 
 
Global Search Configuration: Homepage Search - Catalog, Knowledge, People 
Item Configuration: Sharepoint External Search File Main Item  
Application: SharePoint Search Connector 
 
Global Search Configuration: Homepage Search - Catalog, Knowledge, People 
Item Configuration: Sharepoint External Search HTML Main Item 
Application: SharePoint Search Connector 
 
Global Search Configuration: Homepage Search - Catalog, Knowledge, People 
Item Configuration: Sharepoint External Search Pages Main View 
Application: SharePoint Search Connector 
 
Global Search Configuration: Homepage Search - Catalog, Knowledge, People 
Item Configuration: Sharepoint External Search PDF Main Item 
Application: SharePoint Search Connector 
 
Global Search Configuration: Homepage Search - Catalog, Knowledge, People 
Item Configuration: Sharepoint External Search PPT Main Item 
Application: SharePoint Search Connector 
 
Global Search Configuration: Homepage Search - Catalog, Knowledge, People 
Item Configuration: Sharepoint External Search Site Main Item 
Application: SharePoint Search Connector 
 
Global Search Configuration: Homepage Search - Catalog, Knowledge, People 
Item Configuration: Sharepoint External Search Word Main Item 
Application: SharePoint Search Connector 

Here is how the search results look like:

[rcard11]

This is an awesome guide!!  Are you able to provide any screenshots or video of what the end user experience is when searching and viewing a result that comes from Sharepoint?  Are they directed to Sharepoint or do they view the article in the portal still?  Thank you!!

[Pavan Teja]

Hi @rcard11 ,

Here is a screenshot of the search results. The results open in the SharePoint url in the same tab. If you want the results to open in a new tab, you need to click on the new tab icon in the right.

[rcard11]

Thanks Pavan!  One more question.  Is it possible to show the "match" from the article within the results so it looks similar to a normal KB article search result?

[Vineet Yadav1]

Hi @Pavan Teja @SmritiGupta 

Great article.
Does this connector support only public access document ?
Does it support document security using 0365 groups.
For example if one pdf file is granted access by using 0365 groups. 

Regards,

Vineet

[Pavan Teja]

@rcard11  - Could you please help me understand match from article with an example?

@Vineet Yadav1 - The results are based on the O365 permissions. Before surfacing results we check if user has access or permission to view that particular file and only then the result is shown.

[rcard11]

Hi @Pavan Teja !  Something like below.  Where it highlights the section of the article or attachment that contains the search term.  

[david_legrand]

Thanks team, I have a remark and maybe some questions.

1) The Script include "sn_sp_con.sharePointClient" used for teh sharepoint utility page (https://instance.service-now.com/nav_to.do?uri=sys_script_include.do?sys_id=e80359dd77f61110575a7aef... isn't aligned with the documentation

Function validateConnectionsRest

//var connectionAlias = 'MicrosoftSharepointOnline';
var connectionAlias = 'MicrosoftSharePointOnline';

Function validateEntityScopeGraph

//var oAuthScopegraph = 'offline_access https://graph.microsoft.com/.default';
var oAuthScopegraph = 'offline_access https://graph.microsoft.com/Sites.ReadWrite.All';

2) Thumbprint

Do we agree, we are talking of the thumbprint property of the ".cer" file that we need to transform in base64?

3)  Sharepoint Ingest User

"Ensure that the Sharepoint Ingest User has the ais_external_content role assigned to act as an integration user." Do we agree this is a "sys_user" record with the role "ais_external_content"?

I'll probably have few more questions but right now, I saw an OAuth error related to the certificate so I'm askign the Azure team to check.

[Pavan Teja]

@rcard11 - Currently the search results also include results based on match from within the article. Example if the search term is parental leaves and document title has leave policy and the doc contains the term parent leaves, it would show up in the search results.

[Pavan Teja]

Hi @david_legrand ,

1. We will update the documentation.
2. Yes, you need to transform thumbprint property to transform in base64.
3. Yes, correct it is the sys_user record with ais_external_content role.

[Tracey6]

Hi @Pavan Teja Does this plugin come with the ISTM Pro?

[david_legrand]

THanks @Pavan Teja , please do because I think everything is configured properly but I can't make it work somehow when indexing and I don't see any error logs.

We are contacting ServiceNow PS on the matter.

[csinnett]

@SmritiGupta Thanks so much for sharing this! We are looking at setting up a graph connector for knowledge article search within SharePoint. Do you know if there is any information available on how search results are prioritised in combination with other search results? For example if a SharePoint search returned a combination of SharePoint pages, ServiceNow knowledge articles and documents, how would these results be prioritised in terms of order in the search results list?

Thanks,

Charlotte.

[Alliv]

Thanks for sharing this guide, very helpful!

I am wondering how the thumbprint should be encoded.
- is it the SHA-1 fingerprint or the SHA-256?
- with or without spaces or colons, or perhaps without any delimiter? (eg. 12:34:56 or 12 23 56 or 123456)

Or maybe I am misunderstanding what the thumbprint is.

Thanks,

Andreas

[JoshuaKim1232]

Hi @SmritiGupta Is it mandatory to set up the SharePoint connector before any results show up from Employee center? 

I have managed to set up all the steps you have provided and even created a sharepoint connector but I cannot see any sharepoint results from search. 
Thank you!

[muhamali1]

Hi @SmritiGupta Great article and thanks for sharing. I have a question that How to index only most current version & do not return archived docs? for example  I need to be able to bring in only SharePoint documents upon search that include the most current document so that employees who use the documents can be confident that they are using the most current.

Thanks in advance to anyone that can help.

[Pavan Teja]

@Alliv - Thumbprint is encoded in Hex by default. Should be converted to Base64.

@JoshuaKim1232 -  Could you please confirm if the created connector state is in Complete

[Pavan Teja]

@muhamali1 - Could you please try using Application Permissions instead of Delegated Permissions for Microsoft Graph while setting up the permissions for the application on the azure portal.

[muzhda]

@SmritiThanks for sharing these instructions. In setting up our SP integration for AI Search on the Employee Center we're coming across an issue with the roles required to be granted in the Azure configuration. We've been informed by our M365 admins that these roles grant access to all SP content across our organization and even though we are only targeting a handful of SP sites for the AI Search, there are concerns with granting roles to the SN Platform that will give access to all SP sites content across our organization because some sites contain company confidential information.

Is there anyway to setup the integration using the SP Integration spokes available oob without granting the SN platform full access to all SP Sites (i.e. only access to select sites).

[SmritiGupta]

Hi @muzhda , Please wait for our February release to be out (which will be out very soon on 1st Feb) for this concern to be addressed. Once you upgrade to the feb release, you as an admin will be able to set permissions for read only or full control for the sites/URL that you index, giving you flexibility to exclude some sub-sites as well. Please look out for our February 2024 release blog for additional details which will be out on 1st Feb 2024. 

[david_legrand]

@SmritiGupta 

Could the team perform a nowacademy video of how to setup showing every single step with a demo Sharepoint environment, not just saying "Here is where you put this information..." but actually doing it, showing what is done on Azure, what is done to create the JWT using OpenSSL

Really everything has to be created, the plugins being activated, the application being created on Azure... while performing the video

It would really help

[Aksha]

Hi @SmritiGupta / @Pavan Teja 

We have configured as per the steps and was able to test graph and sharepoint API successfully as well from ServiceNow. The connector post indexing is in complete state. Still no results are showing in the portal because the Sharepoint indexed external content table seems to be empty. Any inputs?

[Suheb Shaik]

Hi @Pavan Teja / @SmritiGupta ,

I appreciate you providing thorough instructions; people who are attempting it for the first time will find them very useful.

Since last year, we have been attempting to set this up for our client. We have encountered a number of difficulties along the way, which we were able to overcome with the assistance of HI support [PRB1685327]. However, we are currently facing another problem: SharePoint Online Search Connector is unable to fetch or recognize users from everyone/everyone except external users site permissions, and the majority of our sites have the same level of permissions. This is a pointless feature if it cannot be used on public sites. Do we have a development strategy or a workaround for this?

[CasandraGB]

Hello! Awesome article.

Question, did any of you tested this with the VA?

If so, do you have examples of the experience?

We're trying to have a "Genius Results" kind of experience, so I was wondering if in the VA, it's like in the AI search ESC portal experience as well.

Thanks a lot in advance for your replies!

Regards

Casandra

[Vadla Srimaan]

Hi @SmritiGupta / @david_legrand / @Suheb Shaik ,

I'm not really clear with this particular step. I don't find the "Microsoft SharePoint Online Certificate" record under certificates table. Should we manually create it?

[Sam179]

I have followed all the steps above but getting the error below when trying to Get OAuth Token for rest.

OAuth flow failed. Verify the configurations and try again. Error detail:invalid_request, AADSTS901002: The 'resource' request parameter is not supported. Trace ID: 8af1fe51-3fa4-4425-b6a3-5edcf88d5900 Correlation ID: 1155becd-7453-4ef3-aca2-fc744f80063d Timestamp: 2024-11-06 10:39:49Z

Logs show "OAuth authentication failed for client_credentials grant type flow" error.

Postman gives no error. Any help appreciated.

 

[SajilalM]

Hey @SmritiGupta this documentation does not mention about 1) Sharepoint Ingest user and its use case whether it is a pre requisite to the search working and how the integration user will be used or this step can be skipped in the Configuration. 2) Also missing the demo of Index sites for search part on how to get the troubleshooting addressed. I am currently facing an error and the logs are not that helpful ' java.net.UnknownHostException: graph.microsoft.come, errorCode:1'

Also one more query if you may answer, i dont see the current capability of the connector working around in Workspace Recommendation Search or Backend Global Search. Is it only restricted to ESC & Service Portal.

Thanks,

Saji

[SaurabGidwani]

Hi @SmritiGupta and @Pavan Teja  Can you guys pls confirm if this will work for OnPrem Sharepoint as well or just Sharepoint online? can I use mid server for On prem?

[balakrishna]

We have integrated AI search with Sharepoint using Sharepoint connector by following your steps. When we are trying to add sites and do indexing via Sharepoint connectors only some of the sites are pulling data.
Any suggestions ?

[SajilalM]

Is there a step for articles to be visible in ESC Portal, the indexing is completed but the articles is still not visible on ESC portal while searching ?

Thanks

[jtshone]

Step 3

"SharePoint Ingest User" record is not available

Does anyone know what we've done wrong?

[Vadla Srimaan]

Hi @jtshone , you need create a user record and add in the new sharepoint Ingest User

@SajilalM , add AI search source in search profile and the publish it.

[SajilalM]

@Vadla Srimaan Yeah those steps are all done thats not the root cause

[jtshone]

Are these instructions still valid?  the instructions and video content are very confused...

a) why is the user in the video in the Employee Center Core scope when no mention of scope in instructions?

b) there is confusion on the redirect url.  Instructions above state 

• Redirect URL: Use the SharePoint instance URL followed by oauth_redirect.do 

but in the video the instructions show the servicenow url and instructions on following also state servicenow instance on Redirect URL:

Graph connection form
REST connection form

Can anyone confirm as i am not confident about the video here......

[Snow62]

Hi @SmritiGupta ,
the Video recording can be found here link showing expired. 
can you provide these links of videos?? it would be helpful.

Thanks

[SmritiGupta]

Hey @Snow62 we recommend you to explore the External Content Connector since the SharePoint Online Search Connector has been deprecated. 

[balakrishna]

Hi @SmritiGupta , Is there any content created for External Content Connector same as SharePoint Online Search Connector for end to end setup ?

[Kass3m]

@SmritiGupta   @pavanteja  can you attach a new link to the videos? because the link is expired.

is Microsoft Sharepoint Spoke also needed as a dependency for this connector? or can we do it without the sharepoint search without the spoke?

[Kass3m]

in the azure configuration piece, point 5, is the certificate .jks being generated and uploaded in Azure?

will this be used in Servicenow in later steps?

[Kass3m]

@Vadla Srimaan Did you figure out the answer to the certificate question?

[Kass3m]

@balak i have some questions on the setup: can we connect?

[Kass3m]

@davidlegrand Did you get these questions answered? I have the same questions

[Vadla Srimaan]

Hi @Kass3m , Yes