Do the thumbs up in prod affect LLM understanding

sofiahd
Kilo Contributor

Hi all,

 

my team is currently testing the new patch 6 for NowAssist. Thumbs up/down seem to be training the LLM in Dev2.  Something I'm wondering though is if the training thumbs up/down in prod will affect LLM training also meaning that if users are not clear on procedures or dislike a procedures may affect 'the truth' of the LLM.

potentially is that something that can be kept for dev2 for training purposes and have thumbs up/down show in prod but only as feedback that don't affect LLM training?

 

6 REPLIES 6

rpriyadarshy
Tera Guru

Hi

 

Thumbs Up & Down are part of feedback loop in terms of LLM response.

 

====================================================================================

How does user feedback from thumbs-up or thumbs-down get used?
Thumbs-up/down feedback provides a high-level signal to help ServiceNow evaluate how generative AI models are performing across different skills and use cases. This input is used to identify general patterns or recurring issues that may inform future improvements.

However, this feedback:

Is not used for automated model retraining or fine-tuning
Does not lead to immediate, instance-specific changes
Does not guarantee resolution of any specific issue
Instead, it is reviewed by ServiceNow’s product and engineering teams to help prioritize model improvements over time—for example, identifying common points of confusion, formatting issues, or gaps in understanding that may affect multiple customers.

Feedback is only collected from customers who have opted into the Advanced AI & Data Product Terms data sharing program. When paired with usage logs and context, this feedback becomes more meaningful and helps our teams better understand and improve model behavior.

==========================================================================

How it goes in background its as below.

 

Have a look at This Table--> sys_gen_ai_log_metadata.

 

Data and Logs are too much encapsulated so not easy to decode . Have a look at table - SYS_GEN_AI_LOG_METADATA

 

rpriyadarshy_0-1755324582071.png

 

 

Just tried Another Like - and 1 dislike in Chat Window and another entry appears for Both

 

rpriyadarshy_1-1755324582105.png

 

 

rpriyadarshy_2-1755324582165.png

 

 

Ane here are the entries for all the feedback.

2 like and 1 dislike 

 

rpriyadarshy_3-1755324583152.png

 

Just to Add here:-

 

 

Regards

RP

++ More Details