Search query parameter evaluation framework
AI Search performs A/B test evaluations that compare result relevancy for alternate sets of search query parameter values. These evaluations determine the models that AI Search uses for machine learning relevancy and for Q&A Genius Results.
The search query parameter evaluation framework enables A/B relevancy testing on live search traffic. AI Search considers results from this live testing along with its offline evaluation of aggregated search signal data when making the following determinations:
- Determining which relevancy model to publish for a search profile. For details on generation and publication of relevancy models, see Machine learning relevancy in AI Search.
- Determining which machine reading comprehension (MRC) model to use when validating potential Q&A Genius Result answers. For more information on the MRC model used for Q&A Genius Results, see Q&A Genius Results.
AI Search computes A/B testing evaluation results on a nightly basis.
Note:
The search query parameter evaluation framework does not support user configured A/B testing. All A/B testing is automatically performed by AI Search as part of its evaluation of machine learning relevancy models.
The search query parameter evaluation framework is part of the Adaptive Telemetry suite of features.
Search query parameter evaluation framework tables
The search query parameter evaluation framework for AI Search uses tables to store results and metrics from A/B testing of live search traffic. AI Search uses the stored data when determining which relevancy model and Q&A Genius Result answer validation model to publish for a search application.
A/B Testing Evaluation table
The A/B Testing Evaluation [evaluation] table stores details for A/B testing evaluations performed on live search traffic.
Note:
The system automatically cleans this table, removing records for inactive evaluations that are more
than two years old.
| Column | Description |
|---|---|
| Number | Autogenerated number to identify the evaluation. |
| Name | Name and description for the evaluation. |
| Source | Name of the table containing records to run the evaluation for. |
| Testing Scope | Scope of the evaluation.
|
| Artifact Provider | List of artifact filtering classes that provide records for testing. |
| Selected Artifacts | Comma-separated list of record sys_id values. When Testing Scope is set to SELECTED, the evaluation only runs against records with sys_id values specified in this list. |
| Minimum Signals for Evaluation | Minimum number of search signals the evaluation should collect before ending.
For details on how this condition interacts with the Minimum Days for Evaluation condition, see End Criteria Operator. |
| Minimum Days for Evaluation | Minimum number of days the evaluation should be active before ending.
For details on how this condition interacts with the Minimum Signals for Evaluation condition, see End Criteria Operator. |
| End Criteria Operator | Logical operator to use for evaluation end conditions when both Minimum Signals for Evaluation and Minimum Days for Evaluation conditions exist.
|
A/B Testing Evaluation Execution table
The A/B Testing Evaluation Execution [evaluation_execution] table stores details of individual operations executed as part of A/B testing evaluations for live search traffic.
| Column | Description |
|---|---|
| Number | Autogenerated number to identify the evaluation operation. |
| Evaluation | Reference to the record for the evaluation on the A/B Testing Evaluation [evaluation] table. |
| Artifact ID | Sys_id for the record analyzed by this evaluation operation. |
| State | State for the record evaluation execution.
|
| Start Date | Date and time when the evaluation operation started. |
| End Date | Date and time when the evaluation operation ended. |
A/B Testing Evaluation Parameter table
The A/B Testing Evaluation Parameter [evaluation_parameter] table stores details of individual search query parameters considered in A/B testing evaluations for live search traffic.
| Column | Description |
|---|---|
| Evaluation | Reference to the record for the evaluation on the A/B Testing Evaluation [evaluation] table. |
| Name | Name and description for the evaluation parameter. |
| Use Artifact's Assigned Parameter |
|
| Parameter Values Traffic Percentage | Percentage of search queries to apply the specified Parameter Values to during the A/B evaluation.
|
| Parameter Values | JSON list of values to compare to the current search query parameter value when Use Artifact's Assigned Parameter is true. |
| Parameter Type | Type for the search query parameter. The selected value determines how the Parameter Values are used during and after an A/B evaluation.
Supported values:
|
| Score Calculation Type | Form of calculation used to compute search query parameter value scores and find the best value.
Supported values:
|
| Signal Provider | The provider for search signals needed to score the search query parameter. Search Event Signal Provider is the only supported value. |
A/B Testing Evaluation Parameter Result table
The A/B Testing Evaluation Parameter Result [evaluation_parameter_result] table stores calculation results for individual search query parameters considered in A/B testing evaluations for live search traffic.
| Column | Description |
|---|---|
| Evaluation Execution | Reference to the record for the evaluation execution on the A/B Testing Evaluation Execution [evaluation_execution] table. |
| Parameter Evaluation | Reference to the record for the search query parameter on the A/B Testing Evaluation Parameter [evaluation_parameter] table. |
| Best Value | Best value for the search query parameter, as determined by the Winning Score. |
| Winning Score | Numeric score for the search query parameter, determined using the Score Calculation Type. |
| Score Metadata | Metadata from the score computation for the search query parameter. |
Exclude a search profile from the search query parameter evaluation framework
Exclude a search profile from A/B testing evaluations of live AI Search traffic. This procedure prevents AI Search from using A/B testing evaluation results when publishing the search profile's search result relevancy model and its Q&A Genius Result answer validation model.
Before you begin
Role required: ais_admin
About this task
The search query parameter evaluation framework performs A/B testing evaluations of search configuration settings using live search traffic. By default, AI Search evaluates configuration settings for all search profiles.
Search administrators can exclude individual search profiles from the search query parameter evaluation framework. Excluding a search profile from the framework prevents AI Search from performing A/B testing evaluations for live search traffic that uses the excluded search profile.
Note:
When you exclude a search profile from evaluations, AI Search no longer considers evaluation results when updating the machine learning relevancy and machine reading comprehension (MRC) models for that search profile. As a result, relevancy scoring
settings and Q&A Genius Result answer filtering settings for the search profile may be less reflective of your search traffic. For more information on how AI Search uses A/B evaluation testing results when publishing these models, see Machine learning relevancy in AI Search and Q&A Genius Results.
Procedure
Result
AI Search no longer performs A/B testing evaluations for traffic that uses the excluded search profile. Machine learning relevancy no longer updates the relevancy model for the search profile.