Encoder - Global

The Encoder API provides a scriptable object used in Predictive Intelligence stores. This object converts input data into vectors of numbers, based on encoder-specific goals and configurations. Encoders can be used independently to run encodings or can be configured as part of solutions to encode text columns.

This API requires the Predictive Intelligence plugin (com.glide.platform_ml) and is provided within the sn_ml namespace.

Encoders are text processing objects that are either pre-trained or trained based on the language datasets you provide. You can train encoders that determine how the system interprets and processes text fields. For ML solutions that include text, you can train an encoder to specify how to process text and use the trained encoder in a solution.

Encoders have configuration and versions, and can be trained independently with their own retraining frequency. API-defined encoders are different from UI-defined encoders, because the retraining of UI-defined encoders is controlled by the solutions using them.

The encoder setup-to-training flow is as follows:

1. Create one or more datasets using the DatasetDefinition API.
2. Use the constructor to create an encoder object.
3. Add the encoder object to the encoder store using the EncoderStore - add() method.
4. Train the encoder using the submitTrainingJob() method. This creates a version of the object that you can manage using the EncoderVersion API.

Once you have trained an encoder, you can use it in a solution object:

• ClassificationSolution
• ClusteringSolution (required unless using the Levenshtein distance algorithm)
• RegressionSolution
• SimilaritySolution (required)

Note:

This API runs with full privileges before the Vancouver Patch 7 Hotfix 2b and Washington DC Patch 7 releases. With later releases, grant access using ACLs. For more information see Query ACLs.

For usage guidelines, refer to Using ML APIs.

Encoder - Encoder(Object config)

Creates an encoder.

To get an encoder for one or more datasets, use this constructor to create a new encoder object with a unique name.

Table 1. Parameters

Name	Type	Description
config	Object	JavaScript object containing configuration properties of the encoder. { "algorithmConfig": {Object}, "datasets": [Array], "domainName": "String", "label": "String", "minRowCount": "String", "processingLanguage": "String", "stopwords": [Array], "trainingFrequency": "String" }
config.algorithmConfig	Object	Optional. JavaScript object containing algorithm configuration properties. 'algorithmConfig' : { "algorithm": "String" }
config.algorithmConfig.algorithm	String	Name of the algorithm for training this encoder. Possible values: paravec: Paragraph vector word embedding. tf-idf: Term Frequency–Inverse Document Frequency (TF-IDF)-based text.
config.datasets	Array	List of DatasetDefinition object names.
config.domainName	String	Optional. Domain name associated with this dataset. Default: Current domain, for example, "global".
config.label	String	Identifies the prediction task.
config.minRowCount	String	Optional. Minimum number of records required in the dataset for training. Default: 10000
config.processingLanguage	String	Optional. Processing language in two-letter ISO 639-1 language code format. Default: "en"
config.stopwords	Array	Optional. Preset list of strings that the system automatically generates based on the language property setting. For details, see Create a custom stopwords list. Default: English Stopwords
config.trainingFrequency	String	The frequency to retrain the model. Possible values: every_30_days every_60_days every_90_days every_120_days every_180_days run_once Default: run_once

The following example shows how to create an encoder job and add it to the encoder store.

var myPrbData = new sn_ml.DatasetDefinition({
    'tableName' : 'problem',
    'fieldNames' : ['short_description'],
    'encodedQuery' : 'activeANYTHING'
});

var myIncidentData = new sn_ml.DatasetDefinition({
    'tableName' : 'incident',
    'fieldNames' : ['short_description', 'description'],
    'encodedQuery' : 'activeANYTHING'
});

var myEncoder = new sn_ml.Encoder({
    'label': "encoder",
    'datasets' : [myPrbData, myIncidentData],
    'algorithmConfig' : {
        'algorithm' : 'tf-idf'
    }
});
var myEncoderName = sn_ml.EncoderStore.add(myEncoder);

Encoder - cancelTrainingJob()

Cancels a job for a encoder object that has been submitted for training.

Table 2. Parameters

Name	Type	Description
None

Table 3. Returns

Type	Description
None

The following example shows how to cancel an existing training job.

var myEncoder = sn_ml.EncoderStore.get('ml_sn_global_global_encoder');

myEncoder.cancelTrainingJob();

Encoder - getActiveVersion()

Gets the active EncoderVersion object.

Table 4. Parameters

Name	Type	Description
None

Table 5. Returns

Type	Description
Object	Active EncoderVersion object.

The following example shows how to get an active Encoder version from the store and return its training status.

var mlEncoder = sn_ml.EncoderStore.get('ml_x_snc_global_global_encoder');

gs.print(JSON.stringify(JSON.parse(mlEncoder.getActiveVersion().getStatus()), null, 2));

Output:

{
  "state": "encoder_complete",
  "percentComplete": "100",
  "hasJobEnded": "true"
}

Encoder - getAllVersions()

Gets all versions of an encoder.

Table 6. Parameters

Name	Type	Description
None

Table 7. Returns

Type	Description
Array	Existing versions of an encoder object. See also EncoderVersion API.

The following example shows how to get all Encoder version objects and call the getVersionNumber() and getStatus() encoder version methods on them.

var mlEncoder = sn_ml.EncoderStore.get('ml_x_snc_global_global_encoder');

var mlEncoderVersions = mlEncoder.getAllVersions();

for (i = 0; i < mlEncoderVersions.length; i++) {
gs.print("Version " + mlEncoderVersions[i].getVersionNumber() + " Status: " + mlEncoderVersions[i].getStatus() +"\n");
};

Output:

Version 3 Status: {"state":"encoder_complete","percentComplete":"100","hasJobEnded":"true"}

Version 2 Status: {"state":"encoder_complete","percentComplete":"100","hasJobEnded":"true"}

Version 1 Status: {"state":"encoder_cancelled","percentComplete":"0","hasJobEnded":"true"}

Encoder - getLatestVersion()

Gets the latest version of an encoder.

Table 8. Parameters

Name	Type	Description
None

Table 9. Returns

Type	Description
Object	EncoderVersion object corresponding to the latest version of an Encoder().

The following example shows how to get the latest version of an encoder and return its training status.

var mlEncoder = sn_ml.EncoderStore.get('ml_x_snc_global_global_encoder');

gs.print(JSON.stringify(JSON.parse(mlEncoder.getLatestVersion().getStatus()), null, 2));

Output:

{
  "state": "encoder_complete",
  "percentComplete": "100",
  "hasJobEnded": "true"
}

Encoder - getName()

Gets the name of the object to use for interaction with the store.

Table 10. Parameters

Name	Type	Description
None

Table 11. Returns

Type	Description
String	Name of the encoder object.

The following example shows how to update Encoder dataset information and print the name of the object.

// Update encoder
var myIncidentData = new sn_ml.DatasetDefinition({
   'tableName' : 'incident',
   'fieldNames' : ['category', 'short_description', 'priority'],
   'encodedQuery' : 'activeANYTHING'
});

var eligibleFields = JSON.parse(myIncidentData.getEligibleFields(encoder));

var myEncoder = new sn_ml.Encoder({
   'label': "my encoder",
   'datasets' : [myIncidentData],
   'inputFieldNames': eligibleFields['eligibleInputFieldNames'],
   'predictedFieldName': 'category'
});

// update encoder
sn_ml.EncoderStore.update('ml_x_snc_global_global_my_definition_4', myEncoder);

// print encoder name
gs.print('Encoder Name: '+myEncoder.getName());

Output:

Encoder Name: ml_x_snc_global_global_my_definition_4

Encoder - getProperties()

Gets solution object properties.

Table 12. Parameters

Name	Type	Description
None

Table 13. Returns

Type	Description
Object	Contents of the Dataset and Encoder() object details in the EncoderStore. { "algorithmConfig" : {Object}, "datasetsProperties": [Array], "domainName": "String", "label": "String", "name": "String", "processingLanguage": "String", "scope": "String", "stopwords": [Array], "trainingFrequency": "String" }
<Object>.algorithmConfig.algorithm	Name of the algorithm for training this encoder. Possible values: paravec: Paragraph vector word embedding. tf-idf: Term Frequency–Inverse Document Frequency (TF-IDF)-based text. Data type: String.
<Object>.algorithmConfig	Optional. JavaScript object containing algorithm configuration properties. 'algorithmConfig' : { "algorithm": "String" } Data type: Object.
<Object>.datasetsProperties	List of DatasetDefinition() properties associated with the encoder. { "encodedQuery": "String", "fieldDetails": [Array], "fieldNames": [Array], "tableName": "String" } Data type: Array.
<Object>.datasetsProperties.tableName	Name of the table for the dataset. For example, "tableName" : "Incident". Data type: String.
<Object>.datasetsProperties.fieldNames	List of field names from the specified table as strings. For example, "fieldNames" : ["short_description", "priority"]. Data type: Array.
<Object>.datasetsProperties.fieldNames.fieldDetails	List of JavaScript objects that specify field properties. [ { "name": "String", "type": "String" } ] Data type: Array.
<Object>.datasetsProperties.fieldNames.fieldDetails.<object>.name	Name of the field defining the type of information to restrict this dataset to. Data type: String.
<Object>.datasetsProperties.fieldDetails.<object>.type	Machine-learning field type. Data type: String.
<Object>.datasetsProperties.fieldDetails.encodedQuery	Encoded query string in the standard platform format. See Encoded query strings. Data type: String.
<Object>.domainName	Domain name associated with this dataset. See Domain separation and Predictive Intelligence. Type: String
<Object>.label	Identifies the prediction task. { "label": "my first prediction" } Data type: String.
<Object>.name	System-assigned name. Data type: String.
<Object>.predictedFieldName	Identifies a field to be trained for predictability. Data type: String.
<Object>.processingLanguage	Processing language in two-letter ISO 639-1 language code format. Data type: String.
<Object>.scope	Object scope. Currently the only valid value is global. Data type: String
<Object>.stopwords	Optional. Preset list of strings that the system automatically generates based on the language property setting. For details, see Create a custom stopwords list. Data type: Array.
<Object>.trainingFrequency	The frequency to retrain the model. Possible values: every_30_days every_60_days every_90_days every_120_days every_180_days run_once Default: run_once Data type: String.

The following example gets properties of an encoder object in the store.

var myEncoder = sn_ml.EncoderStore.get('ml_sn_global_global_encoder');

gs.print(JSON.stringify(JSON.parse(myEncoder.getProperties()), null, 2));

Output:

*** Script: {
  "datasetsProperties": [
    {
      "tableName": "incident",
      "fieldNames": [
        "assignment_group",
        "short_description",
        "description"
      ],
      "encodedQuery": "activeANYTHING"
    }
  ],
  "domainName": "global",
  "label": "my encoder definition",
  "name": "ml_x_snc_global_global_my_encoder_definition",
  "processingLanguage": "en",
  "scope": "global",
  "stopwords": [
    "Default English Stopwords"
  ],
  "trainingFrequency": "run_once"
}

Encoder - getVersion(String version)

Gets an encoder by provided version number.

Table 14. Parameters

Name	Type	Description
version	String	Existing version number of an encoder.

Table 15. Returns

Type	Description
Object	Specified version of the Encoder() object on which you can call EncoderVersion API methods.

The following example shows how to get the training status of an encoder by version number.

var mlEncoder = sn_ml.EncoderStore.get('ml_x_snc_global_global_encoder');

gs.print(JSON.stringify(JSON.parse(mlEncoder.getVersion('1').getStatus()), null, 2));

Output:

{
  "state": "encoder_complete",
  "percentComplete": "100",
  "hasJobEnded": "true"
}

Encoder - setActiveVersion(String version)

Activates a specified version of an encoder in the store.

Table 16. Parameters

Name	Type	Description
version	String	Name of the Encoder() object version to activate. Activating this version deactivates any other version.

Table 17. Returns

Type	Description
None

The following example shows how to activate an encoder version in the store.

sn_ml.Encoder.setActiveVersion("ml_incident_categorization");

Encoder - submitTrainingJob()

Submits a training job.

Note:

Before running this method, you must first add an encoder to the store using the EncoderStore - add() method.

Table 18. Parameters

Name	Type	Description
None

Table 19. Returns

Type	Description
Object	EncoderVersion object corresponding to the Encoder being trained.

The following example shows how to create a dataset, apply it to an encoder, add it to a store, and submit the training job.

// Create a dataset 
var myData = new sn_ml.DatasetDefinition({

  'tableName' : 'incident',
  'fieldNames' : ['assignment_group', 'short_description', 'description'],
  'encodedQuery' : 'activeANYTHING'

});

// Create an encoder 
var myEncoder = new sn_ml.Encoder({

  'label': "my encoder definition",
  'datasets' : [myData],
  'predictedFieldName' : 'assignment_group',
  'inputFieldNames':['short_description']

});

// Add the encoder to the store to later be able to retrieve it.
var my_unique_name = sn_ml.EncoderStore.add(myEncoder);

// Train the encoder - this is a long running job 
var myEncoderVersion = myEncoder.submitTrainingJob();