How are you measuring true Virtual Agent, Now Assist resolution effectiveness beyond case deflection

[Osvaldo Santos]

Hello Community,

We're currently evaluating the effectiveness of Now Assist Virtual Agent and are looking to move beyond traditional case deflection metrics.

While measuring the number of incidents or requests deflected is relatively straightforward, we're struggling with a more important question:

How do you validate that a conversation actually resulted in a successful resolution for the employee, rather than simply providing guidance or ending the interaction without a confirmed outcome?

Specifically, I'm interested in learning:

• What KPIs do you use to measure Virtual Agent / Now Assist success beyond case deflection?
• How do you distinguish between:
  • Deflected and resolved
  • Deflected but unresolved
  • Deflected but later escalated to an agent or ticket
• Are you using conversation analytics, user feedback, follow-up surveys, behavioral signals, or other methods to confirm resolution?
• Do you measure "Resolution Rate" separately from "Deflection Rate"?
• Has anyone implemented a mechanism to track whether the user completed the intended action after receiving guidance from the Virtual Agent?
• What executive metrics have proven most credible when demonstrating business value and employee experience improvements?

Our concern is that a high deflection rate alone may overstate success if users leave the conversation without actually solving their issue.

I would appreciate hearing how other organizations are approaching this challenge, especially those using Now Assist, Virtual Agent Analytics, or User Experience Analytics (UXA) to measure real outcomes.

Thanks in advance for sharing your experience and best practices.