In what table can I find user utterances?

[Chris D]

For the longest time, I've been using the interaction_log table to quickly and simply query real user utterances - i.e. "how many people typed in "x" to VA?" - but I'm finally asking for help after being frustrated by the 40 character limit on the Utterance field on that table which truncates many utterances.

I'm sure one option is to simply expand the max length on that table - no idea why it's so small ootb - but that only addresses data going forward and still I have to imagine that there's a better ootb table for this real data.

I'm aware of the open_nlu_predict_intent_feedback table which is incredibly valuable in debugging user utterances and their predictions - but the problem with this table is that individual utterance entries are not unique. i.e. if one utterance results in two predictions, there will be two entries in the table.

For this scenario I'm asking about, I just want to know each utterance that's used - I don't care about the predictions nor do I want to do data manipulation/exports to cull the non-unique entries. This may be a really simple and obvious question, but what table gets me this data?

[sachin_namjoshi]

You can find utterances in open_nlu_predict_log table.

Regards,

Sachin

[Chris D]

Ahhhh... this seems like really great info... but it only goes back 37 days for some reason and therefore is missing a ton of historical data.

We've been on Quebec for almost a year now, but I even see this table referenced in the Orlando documentation here - Debug a Virtual Agent topic | ServiceNow Docs - so it doesn't seem like something new added.

That article doesn't mention anything about the table being automatically cleaned up either - do you know why historical data is missing? Does your table go back all the way to when you implemented NLU??

[sachin_namjoshi]

This depends on table rotation of open_nlu_predict_log table on your instance.

This open_nlu_predict_log extends system log table.

Log tables and it's extended tables are table rotated for performance reasons.

You should be to find your old data in one of these rotated shards.

If not, I suggest to create a HI support case to get this data.

[Chris D]

Ah that explains it. Thank you.

This log table being rotated is a killer though. What's bizarre is that all utterances (albeit with the 40 character limit) are in the interaction_log table (not extended from the Log Entry table) - that's 100k+ for us (Utterance is not empty) in the past two years. So I would expect some comparable ~100k entries in my open_nlu_predict_log table - if it were not rotated of course. As far as tables goes - especially log tables - that's not even a lot. And ironically, this interaction_log table - which isn't rotated - has much more than just the utterances, so it actually has 1.2+ million records.

You make it seem like we can reach out to ServiceNow to get a complete history of this log - and thank you for that idea - but what I'm looking for isn't a single time extract. I want to be able to do quick queries and even exports on the data so I can improve my NLU model based on real utterances.

Do you have any advice for this?