Metrics for measuring Service Portal user engagement

ElizaDC · ‎09-20-2016

As we draw near deploying Service Portal within our organization, our team is trying to come up with metrics on how Service Portal has been accepted by our users and how engaged are they into utilizing the portal. Any metric suggestion and recommendation is much appreciated!

Erich Zirnhelt · ‎09-21-2016

Hi Eliza,

the metrics are not available in the product, so we built the logic and data model ourselves.

Basically, certain events, like chosen clicks, pageloads and searches, write to a tracking table we defined. We've embedded the capture code into all the pages and widgets we need to track.

The records we write include such data as:

Session ID - so we can tie all the events to the same session
User ID - so we can count unique users, or dot-walk to user-related data
Browser/OS - for understanding our users' technology
Current/Previous page - for understanding user paths
Record Count - since the writes occur asynchronously, we can't always rely on the timestamps to determine the order in which things happen - this incrementing counter lets us order the steps correctly
Other data like search strings, the nature of items that were clicked (e.g., they refer to content off the instance)

We havent yet fully exploited all the data we are capturing, but Ian did list some of the ways we are.

Let us know if you want to know anything else!

View solution in original post

amywei · ‎09-15-2017

Thanks Erich. I will take a look of the two tables you mentioned. I was able to pull up sp_log (Service Portal Log Entries) but not the other table. can you tell me what is ua_sp_user_session for?

thanks,

Amy

Michael QCKM · ‎11-29-2018

zirnhelt,

We would also REALLY appreciate this information.

I as a Knowledge Admin only, and not a SN Admin, cannot see or access the Service Portal Dashboard, nor anything involving the sp_log. I've asked the in-house team and they say it cannot be shared.

We have ITIL users able to access the Self Help "Knowledge" Knowledge Base
ESS users obviously also access the same Self Help "Knowledge" Knowledge Base .

As a Knowledge Base Administrator or Admin, we'd like to run a Report or Dashboard to SHOW and PROVE how ESS users are NOT creating Incidents about issues because they are searching the Self Help "Knowledge" Knowledge Base and viewing articles resolving their issues. This is deflection. The NOT-Creation of Incidents because of SUCCESSFUL, fruitful Knowledge Searches.

One requirement is to filter out ITIL users so we don't muddy up the data.
Another Requirement is to identify search strings/terms that are helpful in finding resolutions
Another Requirement is to identify search strings/terms that are NOT helpful in finding resolutions, so the Knowledge Admins and Authors have actionable data to improve the deflection count.

*I should add that we have not yet merged the Knowledge management Service Portal (/kb) with the Service Portal (/sp), so we are missing out on many of the new features, AND...

*We have type-ahead turned on in the Service Portal, so it is logging hundreds of extra rows per incremental search in the sp_log, making consumption and use of this loge nearly useless.

Erich Zirnhelt · ‎12-18-2018

Hi michaela,

your struggles are common and despite a lot of discussion internally, we haven't solved this well yet. We are working on an approach that would work across service portals and applications for all customers.

Some of the challenges:

How do you define and derive when a user has the intention to create an incident? This will be different across customers' instances.
How do you know when one set of searches is or is not related to another set that occurs some time later? This may be time-based, but more ideally topic-based. Again, this would depend on the customers.
How do you treat the 'softer' deflections, like if the user saw what they needed from a search result snippet?
Does directing the user to the service catalog count as a deflection? Again, customer-specific, and depends on what the subsequent workflow is doing (or how automated it is).

We have a model that is working on HI, but it's built to HI's own use cases and would not work for many customers without some tweaks.

In the short term you may need to build a custom approach. I'm hoping we can have something to offer in the Orlando timeframe.

Michael QCKM · ‎12-18-2018

zirnhelt,

Thanks. Agreed these are tough metrics... as such I've been investigating Journey Analytics. It claims to map the whole user path, but is also complex/costly.

In a previous KB, we'd added a Knowledge Block to all Self Help articles (and a Home Page link) that said "Not finding what you're looking for? Create a ticket" By users clicking that link, the idea was to map the Session of what searches they'd done, article views, and then the subsequent Incident, and link them as one "experience". Our Analytics tool was less than desired, so we did what we could to more closely identify intention/search/views/etc.

I would like to implement and make use of the Faceted Search to help narrow the scope, but we aren't there yet.

We also look to the Agents subsequently attaching a KB article to those specially-created Incidents, based on their work with the customer - in order to identify possible internal articles that could be translated to Self Help.

I see your point about directing to a Catalog item, and have flip-flopped on whether it should be treated as a deflection. Measuring Form use via the Service Portal initial query does reflect Portal value though and lets you know if you've got the right Catalog configuration and keywords for users to find the Form.

As long as ServiceNow is listening, understanding our need to measure ROI of Self Help Portal, and documenting ideas, that is still a positive.

In the meantime, we'll keep our eyes open and brains working on a solution.