Adapting Nemotron Rerank to unstructured enterprise data for better reranking quality and faster serving.
By Orlando Marquez Ayala, Research Scientist, ServiceNow; Sridhar Nemala, Senior Director, ServiceNow; Sean Sodha, Product Manager, NVIDIA; Farzan Memarian, Generative AI Scientist, NVIDIA; and Isabel Hulseman, Product Marketing Manager, NVIDIA
When an employee needs to troubleshoot a system, understand a policy, or follow a workflow, an AI assistant needs to find the information that helps them take the next step. ServiceNow and NVIDIA worked together to improve that retrieval stage by adapting NVIDIA Llama Nemotron Rerank 1B v2—part of the NVIDIA Nemotron Retriever collection— to enterprise data for search, retrieval-augmented generation (RAG), and agentic use cases.
Efficient information retrieval is fundamental to ServiceNow’s AI products such as Zero Touch Service Desk, where autonomous digital workers need to retrieve solution steps as part of self-assigning, diagnosing, and resolving standard issues.
Nemotron Retriever models are a part of the NVIDIA Agent Knowledge Foundation which provides open NVIDIA models and accelerated libraries to help developers connect enterprise agents to relevant evidence, business meaning, and data. ServiceNow’s Core AI team improved reranking quality through fine-tuning and checkpoint merging, then served the model using TensorRT with FP8 precision, achieving better reranking quality and 24.1% lower median reranker latency than BGE Reranker v2 M3, with latency measured on the production server.
Adapting the ranking stage
An initial search can return many passages related to a question. A cross-encoder reranker processes the query and each candidate passage together, scoring their relevance to determine which results come first. For an AI assistant, that ordering helps determine which information becomes the foundation of its response.
Enterprise relevance has to account for specialized terminology, different business domains, and multiple languages. ServiceNow already had domain-specific question-answer pairs and an evaluation framework. Those resources gave the team a way to adapt the reranker and measure the change while holding the first-stage embedding model and candidate depth fixed.
Figure 1. The quality-evaluation flow. The highlighted reranker is the component adapted in this work. Moveworks product ranking also uses other features; this diagram describes the offline evaluation.
Learning relevance from enterprise data
The team started with the open NVIDIA Llama Nemotron Rerank 1B v2 model and NVIDIA’s public reranking model fine-tuning recipe. The recipe provided a workflow for turning domain data into training examples and evaluating an adapted model.
The training mix included retrieval data from different sources and domains, synthetic examples generated from enterprise corpora using the recipe, and multilingual data. Adaptation was sensitive to learning-rate choice and data composition. The team adjusted hyperparameters and broadened its data mix, seeking better relevance on enterprise content while limiting overfitting.
Evaluation covered internal search in the fine-tuning domain, internal search mostly outside that domain, multilingual workloads, and public retrieval benchmarks. Results varied by group: a candidate trained on all available data still lagged on some workloads.
ServiceNow addressed this through a linear merge of two fine-tuned checkpoints. The team combined their model weights and evaluated the merged candidate across its benchmark groups, obtaining a more balanced result across distributions. Training-data composition, hyperparameter tuning, and checkpoint selection all contributed to the adaptation process.
Improving quality over the base model
The comparison used EmbeddingGemma-300M with 768-dimensional embeddings to retrieve the top 100 candidates for each query. ServiceNow measured reranker quality using binary normalized discounted cumulative gain, or binary NDCG, which evaluates ranking with relevant-or-irrelevant labels. Higher scores indicate better ranking.
|
Reranker |
Overall |
Public BEIR subset |
Public MIRACL (multilingual) |
Internal search, in domain |
Internal search, mostly out of domain |
Internal multilingual |
|
BGE Reranker v2 M3 |
0.6428 |
0.6524 |
0.8558 |
0.5448 |
0.3966 |
0.6692 |
|
Base Llama Nemotron Rerank 1B v2 |
0.6615 |
0.6816 |
0.8476 |
0.5860 |
0.3766 |
0.7089 |
|
Fine-tuned Nemotron reranker TRT FP8 |
0.6945 |
0.6982 |
0.8521 |
0.6239 |
0.4067 |
0.7513 |
Binary NDCG results from ServiceNow’s evaluation. All rows use the same first-stage retrieval setup. Overall covers the full evaluation suite, including public benchmarks; it is not the arithmetic mean of the displayed group averages.
Overall binary NDCG rose from 0.6615 to 0.6945: +0.033 absolute, or 3.3 points on a 0–100 scale, and a 5.0% relative improvement over base Nemotron. In-domain search rose from 0.5860 to 0.6239, mostly out-of-domain search from 0.3766 to 0.4067, and multilingual retrieval from 0.7089 to 0.7513.
The group results also show why an aggregate alone is insufficient. The fine-tuned model led across the internal workloads and public BEIR subset, while BGE retained a slightly higher public MIRACL score. Overall, the new production model is 5.17 points higher on a 0–100 scale than the previous BGE model, demonstrating the value of using Nemotron as the base checkpoint and of fine-tuning using enterprise data.
Improving relevance was one part of the development process. The next step was to serve the adapted model efficiently enough for an interactive search experience, bringing ranking quality and response time into the same comparison.
Better reranking with a shorter wait
ServiceNow deployed a fine-tuned NVIDIA Llama Nemotron Rerank 1B v2 in its search pipeline using TensorRT with FP8 precision.
The team tested different configurations under different user loads.
Before deployment, ServiceNow compared reranker latency on an H200 production server that was also the busiest available:
|
Latency percentile |
BGE reranker |
Nemotron FP8 |
Reduction |
|
P50 |
141 ms |
107 ms |
24.1% |
|
P90 |
247 ms |
206 ms |
16.6% |
|
P99 |
637 ms |
585 ms |
8.2% |
Measurements are from ServiceNow’s production reranker service and reflect the compared serving configurations.
Viewed together, the reported quality and latency results show the outcome of adaptation and serving optimization: the fine-tuned Nemotron FP8 configuration combines a higher aggregate quality score with lower median latency than BGE. Figure 2 illustrates the reranker journey in a single quality-versus-latency view, where movement toward the upper left means better quality and a shorter wait. Its horizontal axis uses the separately reported worst-case latency, rather than the percentile measurements in the table above.
Figure 2. Quality and latency across the reranker journey. Quality scores are displayed on a 0–100 scale; the horizontal axis shows worst-case latency in seconds. The base-model horizontal position and dashed path are illustrative: a comparable production latency measurement for base Nemotron BF16 is unavailable. The diagram is not a fully measured performance trajectory.
At the median, latency fell 34 ms (141 → 107). The Nemotron FP8 configuration also reduced latency for the slower requests represented by P90 and P99. These service measurements complement the quality results by showing how the adapted model performs in a serving environment.
Bringing the model into product pipelines
The reranker is running in production in ServiceNow search and is being tested in the Moveworks development environment. Moveworks ranking combines model-derived features with metadata such as channels and timestamps, so product-level testing needs to assess how the reranker interacts with the rest of that pipeline.
Within an AI-assistant pipeline, better reranking can help prioritize useful evidence from sources such as knowledge base articles and keep unrelated content out of the top result, potentially reducing incorrect answers, clarification turns, and repeat downstream LLM calls. Evaluating the complete pipeline connects offline relevance gains to the employee’s search experience.
Query: How much can I spend on food when traveling?
Relevant passage: Ya i am trying to see why. Below 3-hour flight spend limit: US 200 USD, UK - 150 GBP
Irrelevant passage: PRIVILEGED AND CONFIDENTIAL If I need to travel for work, what is the process? You need to ask your manager for permission to travel by submitting a request
Opening more enterprise knowledge to AI
Reranking is one part of the journey from a question to useful information. Looking ahead, the new Nemotron 3.5 Embed, Rerank, and Parse models could strengthen that journey, from finding candidate content to identifying the best matches and extracting information from documents.
For ServiceNow, retrieval quality matters well beyond search. It helps determine the more precise and relevant information that AI assistants and agents use to answer questions and take action. This work with NVIDIA strengthens the retrieval foundation behind those experiences, with applications across ServiceNow AI Search, Otto, and the broader enterprise search and agentic AI experiences ServiceNow is building with Moveworks.
By improving which enterprise knowledge is surfaced and how quickly it can be retrieved, this work can help ServiceNow’s Otto provide more precise and relevant information for generating answers or initiating workflows. The result is a foundation for more relevant search, better-grounded AI experiences, and more effective AI agents across the enterprise.
Future evaluations of Nemotron 3.5 Embed and Rerank could explore new ways to find and prioritize enterprise content, including multimodal retrieval. Potential work with Nemotron 3.5 Parse and ServiceNow’s Document Intelligence team could extend that exploration to information inside documents. Those evaluations would help determine where newer models can add value.
Getting started with domain adaptation
“For enterprise search, relevance and latency need to improve together. Our work with ServiceNow shows how fine-tuning and inference optimization can turn a pretrained reranker into a model that better fits both the organization’s content and its serving requirements.”
— Farzan Memarian, Generative AI Scientist, NVIDIA
This collaboration offers a practical sequence for enterprise retrieval teams: establish a baseline on representative data, inspect results across workloads, iterate on training and checkpoint selection, and measure the model in its intended serving environment. Holding the first-stage retriever and candidate depth fixed makes the quality comparison easier to interpret; product-pipeline testing then establishes what that improvement means for users.
Developers can start with the NVIDIA Llama Nemotron Rerank 1B v2 model card and NVIDIA’s reranking model fine-tuning recipe, which covers synthetic data generation, data preparation, training, evaluation, export, and deployment guidance. The goal is to turn an organization’s own knowledge into better retrieval, helping employees find useful information and move their work forward.
