kevinanderson
Giga Guru

So,

We just got done implementing Spanish in our instance, and I want to share with you something that took us many hours to get working correctly - transform maps for sys_translated_text, sys_choice, sys_documentation, sys_translated, and sys_ui_message.

According to SN, the out of box transform maps that come with the language pack plugins are not for use by customers, and are only intended for use by the plug-in process.

So, we had to copy and modify these transform maps to work correctly. We had thousands of records related to our translation data, and using XML export / import was not practical as the files were very large and took many hours to import into the target instance ( we had to break them into 2k row 'chunks' to get them to process at all without timing out ).

I spent several hours on the phone with HI support building these transform maps, and I would like to present them to the community, so that others shall not suffer the same pain we did migrating language data manually.

After building these transform maps, we were able to move ~65k records in only a few minutes.

The key to this is:

1. copy the oob transform maps, and disable the scripts - instead do a direct field mapping

2. copy the coalesce field settings from the oob transform maps fields to the custom transform maps

3. when exporting the data, ensure all the fields in the transform map are in the export

4. add the document key sys_id to the sys_translated_text table to accommodate the transform map (done in attached update set)

5. convert the XLSX to XLS before importing to target system for the transform ( I hear this will be fixed in Jakarta )

6. use the following onbefore script for the document key field on sys_translated text (Thanks to Chris Beltran @snc / Rod Godfrey @ snc for providing this basis for this script)

(function runTransformScript(source, map, log, target /*undefined onStart*/ ) {

  // validate the document key can be built - table exists and target record can be found in the table

  gs.include('j2js');

  var isValidTable = function(table_name) {

                      result = false;

                   

                      if (JSUtil.notNil(table_name)) {

                              var gr = new GlideRecord('sys_db_object');

                              if (gr.get('name', table_name)) {

                                      result = true;

                              }

                      }

                      //gs.log('isValidTable for '+table_name+': '+result, 'transform map - sys_tanslated');

                      return result;

              };

             

  // string trim polyfill

              String.prototype.trim || (String.prototype.trim = function() {

                      return this.replace(/^[\s\uFEFF\xA0]+|[\s\uFEFF\xA0]+$/g, "");

              });

              var _doc = source.u_document.toString();

              var _tbl = source.u_table_name.toString();

                           

              var table_name = (_tbl).trim();

  var process_record = false;

  if (isValidTable(table_name)){

  if (JSUtil.notNil(source.u_document_key_sysid)){

  // search for the document key sys_id in the target table

  var _gr = new GlideRecord(table_name);

  if (_gr.get(source.u_document_key_sysid)) {

  process_record = true;

  }

  else{

  gs.log('Warning: unable to find matching record with sys_id: ' + source.u_document_key_sysid+' in table '+table_name, 'transform map - sys_tanslated');

  }

  }

  else{

  gs.log('Warning: invalid document key sys_id for record: ' + _doc, 'transform map - sys_tanslated');

  }

  }

        else{

  gs.log('Warning: invalid table: '+table_name,'transform map - sys_tanslated');

  }

        // if the document key sysid or the target table not found, skip this record

        if (!process_record){

  ignore = true;

  }

})(source, map, log, target);

5 Comments
kevinanderson
Giga Guru

I updated the script and attached a new update set to correct sys_translated_text.



new transform logic is as follows:


1. document key sys_id field added to sys_translated text


2. exported data must include this column


3. transform map on before script   - validated target table exists, and record exists in target table that has a sys_id matching to document key sys id


4. if onbefore script unable to validate table / sys_id, record skipped in the import set


5. document_key_sys_id field in source data directly mapped to document field in sys_translated_text


robertsduca
Giga Contributor

This is fantastic! I've been struggling for the past week trying to figure out how to create these document IDs so the transform can marry the object and the field.



Here is my use case that I am trying to apply your method above to and was hoping you could help me...



I have 488 Software Catalog Items that need the names and descriptions translated to French Canadian. I have all the translations ready to to go and will have to do it in two waves, once on the cmdb_software_product_model table to translate the descriptions (which should post to the catalog item through the catalog utils) and then again on the catalog items for the catalog item name.



Im stuck on   your export step.. how do I export the data from the cmdb_software_product_model table and get the right sysID to import back in using your custom transform map.


kevinanderson
Giga Guru

One approach is to to add a new column to the table that displays the record's sys_id.   A calculated column does the trick (see images below).



If you don't want to edit a bunch of OOB tables to add the sys_id as a display, you can copy the table and data to a temp table, and then add the calculated column there



Capture_add_calc_column1.PNG



Capture_add_calc_column2.PNG




Capture_add_calc_column3.PNG





Capture_add_calc_column4.PNG


Capture_add_calc_column5.PNG



Capture_add_calc_column6.PNG


robertsduca
Giga Contributor

that is a great suggestion! I would not want to add columns to their OOB table for sure...



As a test last night I did pull the sysID from the sw product model and added it manually to the import set table and it still skipped it on the transform using your custom map, so I still must be doing something wrong:



find_real_file.png



find_real_file.png


pascalfrencken
Mega Sage

kevinanderson Thanks for sharing this. We're currently struggling with getting translations for the 7 languages we support imported into Translated Text. I've downloaded your update set and loaded it, but am struggling with getting an import into Translated Text done properly. Instead of updating the existing records, it inserts new ones. Could you share a quick step-by-step instruction on how we should use the transform map you created?



Best regards,


Pascal Frencken



Update: never mind, I found out how it worked myself, and managed to get the translations imported.