Alex Coope - SN
ServiceNow Employee
ServiceNow Employee
This post has been updated to reflect multiple feature requests I've had since I originally posted this. 

The artifact now searches through multiple different aspects of a portal - Taxonomies, Topics, Menu's, Content Experiences,
Themes, Footers & Headers, Web Applications, Quick Links, Service Catalogs and their Categories as well as
KB Categories, widgets called from other widgets, custom Angular templates, if a variable is sent to getMessage it will check if it's in
a defined variable or if it's being sent from a function it will check for those function calls to find the variables in the calls and more...

A massive thank you to everyone who is providing feedback and suggestions, please keep them coming as you are all
helping to make this better for everyone - it is greatly appreciated by many!

Also, we are working on a better way to release this. Originally it was just a PoC and one to teach people how to make really powerful
artifacts. Now it's turned into something quite a bit more - watch this space...

 

So it seems I caused a bit of a stir with my last webinar at the Employee Center Academy. In that, it's become the single most asked question I now get - "how did you do it?" (referring to localizing the portal into Japanese).

If you haven't seen the session I highly recommend you check it out:

 

In this post I'm going to go through some concepts, some sneak peaks behind the scenes, how I rationalise table hierarchies for solution architecture and even share my prototype Localization Framework artifact script to get you started. 

This one is going to be quite technical and long so make sure you have a coffee at the ready and some snacks on hand because this is going to be a long one!

 

 

I have a portal, it needs to be in other languages

If we look at the new /esc portal, imagine we want to go from pure English:
find_real_file.png

 

To also have another Language (imagine Japanese in my example);
find_real_file.png

In this new portal, we now need to consider the "Mega Menu", "Topics" and even "Quick Links" (the widget title is in English as it's a pre-release version and will be fixed by the time you have access to it).
These new components will be super important, as we'll need to understand what tables they are in and therefore what their respective relationships are in respect to the Portal they are shown on. 

As a recap, you should remember from our training course (available here) that for custom widgets etc, we need to follow certain coding standards to ensure all end-user facing strings are properly externalised. Here's a quick recap slide to show what I mean, which details how text is calling the Message API based on it's string type:
find_real_file.png

 

To test if everything is indeed "externalised" as needed, if we remember from our training, we can leverage the "i18n debugger" which should show us a Prefix for each UI string representing that the system would expect a translation for it in one of the 5 tables;
find_real_file.png

 

If you're a regular reader of my blog, you'll know that whilst I'm a big proponent of the Localization Framework (possibly because I helped design it) I've also shown how you can write your own Artifacts, for example Portal Announcements and Surveys

Tip, if you haven't yet read those posts, please read them both before continuing, because they will act as foundational knowledge for the following aspects.

This then means, did I achieve my objective (in under 2 days I need for prep) by leveraging the Localization Framework? The simple answer is yes. I wrote a prototype Artifact to go and identify everything I could within that given portal - which required mapping out the table hierarchies of the types of records used in this new portal (the way I think about it is, it's a pyramid of data, so knowing how that pyramid is structured is the objective).

 

 

How did I do it?

For sake of time, lets assume we've built to coding standards (meaning everything is correctly externalised), let's consider making our new Artifact. The first consideration we would need to think about is where does a portal's hierarchy start?

The tip would be the [sp_portal] table, then we need to think about all of the known pages directly related to it, then the page route map, then all of the widgets (and their respective instances) in those given pages, as well as the new "mega menu" and it's entries, the new "topics" and their sub-values, as well as the new "footers". 

This therefore means, we need to follow the relationships to see if we can create suitable sub-queries from the [sp_portal]. In the most part we can, but due to the unique nature of how Service Portals are built, we can't necessarily obtain everything this way. And so, in my prototype it's not 100%, but it's pretty close.

As we're working on the Rome release, this artifact will be a "Processor Script" (aka a Script Include) and the UI action will be defined on the [sp_portal] table.

  • To learn all of the necessary steps make sure you read the two blog posts above and make sure you read the Product Documentation here (for what to do) and here (to understand what functions you have available).

Remembering this is a prototype, there's for sure many areas of improvement (optimization and efficiency), however this should cover the vast majority of scenarios. The idea here, is to show you that if you understand the table hierarchy of that proverbial thing you want to translate, then you too can write your own artifact should you need to.

 

 

[Update]
I thought it might be helpful to show how a portal is typically structured to explain the logic I applied to designing this POC artifact:

this is a simplification of what a Portal's structure looks likethis is a simplification of what a Portal's structure looks like

 

 

So here it is (it's quite a long one, you've been warned),

The artifact is called;

AlexCoopeSN_0-1687961785927.png

Important note - as with any Script Include make sure that the name of it is the exact same as the first declared object, in this case 
"LF_PortalProcessor"


With the processor script looking like this;

If you're revisiting this post, you might notice my code example is longer than before. That's because in a recent call with a Customer, they asked if I could also check for the associated Theme, hence the changes.

 

Note - this artifact will pick up a lot of strings. If you are translating into a net-new language, it may be more performant to use the Page level artifact further down below, but please be patient once you've selected your language at the request stage as it may take a couple of minutes to generate the task (even if it doesn't look like it's doing anything).

 

 

 

 

 

var LF_PortalProcessor = Class.create();
LF_PortalProcessor.prototype = Object.extendsObject(global.LFArtifactProcessorSNC, {
    category: 'localization_framework', // DO NOT REMOVE THIS LINE!

    /**********
     * Extracts the translatable content for the artifact record
     * 
     * @Param params.tableName The table name of the artifact record
     * @Param params.sysId The sys_id of the artifact record 
     * @Param params.language Language into which the artifact has to be translated (Target language)
     * @return LFDocumentContent object
     **********/
    getTranslatableContent: function(params) {
        /**********
         * Use LFDocumentContentBuilder to build the LFDocumentContent object
         * Use the build() to return the LFDocumentContent object
         **********/
        var tableName = params.tableName;
        var sysId = params.sysId;
        var language = params.language;
        var lfDocumentContentBuilder = new global.LFDocumentContentBuilder("v1", language, sysId, tableName);
        var pageArr = []; // for later
        var widgetArr = []; // we will need this later

        // what portal are we looking at?
        var portalCheck = new GlideRecord('sp_portal');
        portalCheck.addQuery('sys_id', sysId);
        portalCheck.query();

        if (portalCheck.next()) {
            // processString will look for the (portalCheck.getDisplayValue()) in sys_ui_message table and translated value too, if present
            lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(portalCheck, 'Portal', 'Name');

            // we need to do some specific AI search checks
            if (portalCheck.enable_ais == true) {
                var getSearchTabs = new GlideRecord('sys_search_filter');
                getSearchTabs.addQuery('active=true^search_context_config=' + portalCheck.search_application.sys_id);
                getSearchTabs.query();
                while (getSearchTabs.next()) {
                    lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(getSearchTabs, "Search", "Name");
                }
                // we also need to do some other filters checks
                var getSearchSorts = new GlideRecord('sys_search_sort_option');
                getSearchSorts.addEncodedQuery('active=true^search_context_config=' + portalCheck.search_application.sys_id);
                getSearchSorts.query();
                while (getSearchSorts.next()) {
                    lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(getSearchSorts, "Search", "Name");
                }

                // we also need to check some suggestion reader groups
                var getSuggestions = new GlideRecord('sys_suggestion_reader_group');
                getSuggestions.addEncodedQuery('context_config_id=' + portalCheck.search_application.sys_id);
                getSuggestions.orderBy('order');
                getSuggestions.query();
                while (getSuggestions.next()) {
                    lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(getSuggestions, "Search", "name");
                }

                // we also need to check the search facets
                var getSearchFacets = new GlideRecord('sys_search_facet');
                getSearchFacets.addEncodedQuery('active=true^search_context_config=' + portalCheck.search_application.sys_id);
                getSearchFacets.orderBy('order');
                getSearchFacets.query();
                while (getSearchFacets.next()) {
                    lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(getSearchFacets, "Search", "Name");
                }

                // exceptions
                lfDocumentContentBuilder.processString("Most relevant", "Search", "Name");
                lfDocumentContentBuilder.processString("Was this suggestion helpful?", "Search", "Name");
                lfDocumentContentBuilder.processString("Top result", "Search", "Name");
                lfDocumentContentBuilder.processString("Yes", "Search", "Name");
                lfDocumentContentBuilder.processString("No", "Search", "Name");
            }

            // we need to check through the Theme Header and Footer of this portal
            var themeHeader = '';
            var themeFooter = '';

            if (portalCheck.theme.header) {
                var getHeader = new GlideRecord('sp_header_footer');
                getHeader.addQuery('sys_id', portalCheck.theme.header.sys_id);
                getHeader.query();
                if (getHeader.next()) {
                    themeHeader = getHeader;

                    // let's now get the specifics in the theme's header
                    lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(themeHeader, "Theme", 'Header');

                    lfDocumentContentBuilder.processScript(themeHeader.template, "Theme", "Header - Template");
                    _getMessages(themeHeader.template.toString(), "Theme", themeHeader.getDisplayValue());
                    // we need to do a quick check to see if the "user_profile" widget is being used
                    if (themeHeader.template.includes('user_profile')) {
                        var getUserProfWid = new GlideRecord('sp_widget');
                        getUserProfWid.addQuery('id', 'user-profile');
                        getUserProfWid.query();
                        if (getUserProfWid.next()) {
                            lfDocumentContentBuilder.processScript(getUserProfWid.template, "User Profile", "HTML");
                            _getMessages(getUserProfWid.template.toString(), "User Profile", "HTML");

                            lfDocumentContentBuilder.processScript(getUserProfWid.script, "User Profile", "Server Script");
                            lfDocumentContentBuilder.processScript(getUserProfWid.client_script, "User Profile", "Client Controller");
                        }
                    }

                    lfDocumentContentBuilder.processScript(themeHeader.css, "Theme", "Header - CSS");
                    _getMessages(themeHeader.css.toString(), "Theme", themeHeader.getDisplayValue());

                    lfDocumentContentBuilder.processScript(themeHeader.script, "Theme", "Header - Script");
                    _getMessages(themeHeader.script.toString(), "Theme", themeHeader.getDisplayValue());

                    lfDocumentContentBuilder.processScript(themeHeader.client_script, "Theme", "Header - Client Script");
                    _getMessages(themeHeader.client_script.toString(), "Theme", themeHeader.getDisplayValue());

                    lfDocumentContentBuilder.processScript(themeHeader.link, "Theme", "Header - Link");
                    _getMessages(themeHeader.link.toString(), "Theme", themeHeader.getDisplayValue());
                }
            }

            if (portalCheck.theme.footer) {
                var getFooter = new GlideRecord('sp_header_footer');
                getFooter.addQuery('sys_id', portalCheck.theme.footer.sys_id);
                getFooter.query();
                if (getFooter.next()) {
                    themeFooter = getFooter;

                    // we have some specific /esc footer checks to make
                    if (getFooter.id == "employee-center-footer") {
                        // we need to double-check to see if the EC or EC pro is installed first
                        var checkEC = new GlideRecord('sys_package');
                        checkEC.addEncodedQuery('source=sn_ex_sp^ORsource=sn_ex_sp_pro');
                        checkEC.query();
                        if (checkEC.hasNext()) {

                            // now we need to go through the footer
                            var footerCheck = new GlideRecord('sn_ex_sp_footer');
                            footerCheck.addEncodedQuery('portalLIKE' + portalCheck.sys_id);
                            footerCheck.addQuery('active', 'true');
                            footerCheck.query();
                            if (footerCheck.next()) {
                                lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(footerCheck, 'Footer', 'Footer');

                                // now we need to find the menus for the footer
                                var footerMenuCheck = new GlideRecord('sn_ex_sp_footer_menu');
                                footerMenuCheck.addQuery('footer', footerCheck.sys_id);
                                footerMenuCheck.orderBy('order');
                                footerMenuCheck.query();
                                while (footerMenuCheck.next()) {
                                    lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(footerMenuCheck, 'Footer Menu', 'Footer Menu');

                                    // now we need to find the items in the menus
                                    var footerMenuItem = new GlideRecord('sp_instance_menu');
                                    footerMenuItem.addQuery('sys_id', footerMenuCheck.menu.sys_id);
                                    footerMenuItem.query();
                                    if (footerMenuItem.next()) {
                                        lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(footerMenuItem, 'Footer Menu Item', 'Footer Menu Item');

                                        // now we need to check each sub-item
                                        var footerMenuSubItem = new GlideRecord('sp_rectangle_menu_item');
                                        footerMenuSubItem.addEncodedQuery('active=true');
                                        footerMenuSubItem.addQuery('sp_rectangle_menu', footerMenuItem.sys_id);
                                        footerMenuSubItem.orderBy('order');
                                        footerMenuSubItem.query();
                                        while (footerMenuSubItem.next()) {
                                            lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(footerMenuSubItem, 'Footer Menu Sub-Item', 'Footer Menu Sub-Item');
                                        }
                                    }
                                }
                            }
                        }
                    }

                    // let's now get the specifics in the theme's footer
                    lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(themeFooter, "Theme", 'Footer');

                    lfDocumentContentBuilder.processScript(themeFooter.template, "Theme", "Footer - Template");
                    _getMessages(themeFooter.template.toString(), "Theme", themeFooter.getDisplayValue());

                    lfDocumentContentBuilder.processScript(themeFooter.css, "Theme", "Footer - CSS");
                    _getMessages(themeFooter.css.toString(), "Theme", themeFooter.getDisplayValue());

                    lfDocumentContentBuilder.processScript(themeFooter.script, "Theme", "Footer - Script");
                    _getMessages(themeFooter.script.toString(), "Theme", themeFooter.getDisplayValue());

                    lfDocumentContentBuilder.processScript(themeFooter.client_script, "Theme", "Footer - Client Script");
                    _getMessages(themeFooter.client_script.toString(), "Theme", themeFooter.getDisplayValue());

                    lfDocumentContentBuilder.processScript(themeFooter.link, "Theme", "Footer - Link");
                    _getMessages(themeFooter.link.toString(), "Theme", themeFooter.getDisplayValue());
                }
            }

            // we need to check if a cart widget is used for this portal and if so which
            var cartProp = gs.getProperty('glide.sc.portal.use_cart_v2_header');
            var cartId = '';
            if (cartProp) {
                cartId = 'sc-shopping-cart-v2';
            } else {
                cartId = 'sc-shopping-cart';
            }
            var getCartWidget = new GlideRecord('sp_widget');
            getCartWidget.addQuery('id', cartId);
            getCartWidget.query();
            if (getCartWidget.next()) {
                _getMessages(getCartWidget.template, "Shopping Cart", 'Cart Widget');
                _getMessages(getCartWidget.script, "Shopping Cart", 'Cart Widget');
                _getMessages(getCartWidget.client_script, "Shopping Cart", 'Cart Widget');
                lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(getCartWidget, "Shopping Cart", "Shopping Cart");
            }

            // we need to check for any Taxonomies used on this portal
            var taxPkg = new GlideRecord('sys_package');
            taxPkg.addEncodedQuery('source=com.snc.taxonomy');
            taxPkg.query();
            if (taxPkg.hasNext()) {

                // only progress if the Taxonomy plugins are installed
                var taxCheck = new GlideRecord('m2m_sp_portal_taxonomy');
                taxCheck.addNotNullQuery('active');
                taxCheck.addEncodedQuery('active=true');
                taxCheck.addQuery('sp_portal', portalCheck.sys_id);
                taxCheck.query();

                while (taxCheck.next()) {
                    // we have to use a while incase there is more than one taxonomy associated
                    // now we need to follow the taxonomy's path
                    var getTax = new GlideRecord('taxonomy');
                    getTax.addNotNullQuery('sys_id');
                    getTax.addQuery('sys_id', taxCheck.taxonomy);
                    getTax.query();
                    if (getTax.next()) {
                        lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(getTax, "Taxonomy", "Name");
                    }

                    var taxTopic = new GlideRecord('topic');
                    taxTopic.addNotNullQuery('taxonomy');
                    taxTopic.addQuery('taxonomy', getTax.sys_id);
                    taxTopic.query();
                    while (taxTopic.next()) {
                        lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(taxTopic, "Topic", "Name");

                        // Now we need to see what's connected
                        var conTopic = new GlideRecord('m2m_connected_content');
                        conTopic.addNotNullQuery('topic');
                        conTopic.addQuery('topic', taxTopic.sys_id);
                        conTopic.addEncodedQuery('content_type=07f4b6bfe75430104cda66ef11e8a9a9'); // we're looking for quick links
                        conTopic.query();
                        while (conTopic.next()) {
                            // we must have found a quick link
                            qls = true;
                            var qlRec = new GlideRecord('sp_page');
                            qlRec.addNotNullQuery('sys_id');
                            qlRec.addQuery('sys_id', conTopic.quick_link.page); // this is the page from the quick link
                            qlRec.query();
                            if (qlRec.next()) {
                                lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(qlRec, qlRec.sys_class_name.getDisplayValue(), qlRec.getDisplayName());
                                _getPage(qlRec.sys_id);
                            }
                        }
                    }
                }
            }

            // now we need to find the menu in this portal
            var menuCheck = new GlideRecord('sp_instance_menu');
            menuCheck.addNotNullQuery('sys_id');
            menuCheck.addQuery('sys_id', portalCheck.sp_rectangle_menu);
            menuCheck.query();
            if (menuCheck.next()) {
                lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(menuCheck, "Menu - " + menuCheck.getDisplayValue(), 'Menu');

                // now we have the menu, we need the options in it
                var menuItemCheck = new GlideRecord('sp_rectangle_menu_item');
                menuItemCheck.addNotNullQuery('sp_rectangle_menu');
                menuItemCheck.orderBy('label');
                menuItemCheck.addQuery('sp_rectangle_menu', menuCheck.sys_id);
                menuItemCheck.query();
                while (menuItemCheck.next()) {
                    if (menuItemCheck.label) {
                        // This will look for the (menuItemCheck.label)'s value in sys_ui_message table.
                        lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(menuItemCheck, menuCheck.getDisplayValue(), 'Menu Item');

                        // we need to check if we have "communities" installed or being used
                        if (menuItemCheck.label == "Employee Forums" || menuItemCheck.label == "Community" || menuItemCheck.label == "Forums") {
                            // we now need to loop through some specific records for the Communities actions and notification types
                            var comMessages = new GlideRecord("sn_actsub_activity_type");
                            comMessages.addEncodedQuery('activity_nl_string.messageISNOTEMPTY');
                            comMessages.query();
                            while (comMessages.next()) {
                                // now we need to get the specific message
                                var getComMessage = new GlideRecord('sys_ui_message');
                                getComMessage.addQuery('sys_id', comMessages.activity_nl_string.sys_id);
                                getComMessage.query();
                                if (getComMessage.next()) {
                                    lfDocumentContentBuilder.processString(getComMessage.message.toString(), "Activity Message", "Name");
                                }
                            }
                        }

                        // now we need to process a page if there is one
                        if (menuItemCheck.sp_page.sys_id) {
                            _getPage(menuItemCheck.sp_page.sys_id);
                        }

                        // lets check if there's a sub-menu also
                        var subMenuItemCheck = new GlideRecord('sp_rectangle_menu_item');
                        subMenuItemCheck.addNotNullQuery('sp_rectangle_menu_item');
                        subMenuItemCheck.orderBy('label');
                        subMenuItemCheck.addQuery('sp_rectangle_menu_item', menuItemCheck.sys_id);
                        subMenuItemCheck.query();
                        while (subMenuItemCheck.next()) {
                            lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(subMenuItemCheck, menuCheck.getDisplayValue() + ' - ' + menuItemCheck.getDisplayValue(), 'Sub-Menu Item');

                            // now we need to process a page if there is one
                            if (subMenuItemCheck.sp_page.sys_id) {
                                _getPage(subMenuItemCheck.sp_page.sys_id);
                            }

                            // lets check if there's a sub-sub-menu also
                            var subsubMenuItemCheck = new GlideRecord('sp_rectangle_menu_item');
                            subsubMenuItemCheck.addNotNullQuery('sp_rectangle_menu_item');
                            subsubMenuItemCheck.orderBy('label');
                            subsubMenuItemCheck.addQuery('sp_rectangle_menu_item', subMenuItemCheck.sys_id);
                            subsubMenuItemCheck.query();
                            while (subsubMenuItemCheck.next()) {
                                lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(subsubMenuItemCheck, menuCheck.getDisplayValue() + ' - ' + menuItemCheck.getDisplayValue(), 'Sub-Sub-Menu Item');

                                // now we need to process a page if there is one
                                if (subsubMenuItemCheck.sp_page.sys_id) {
                                    _getPage(subsubMenuItemCheck.sp_page.sys_id);
                                }
                            }
                        }
                    }
                }
            }

            // lets trawl the page route map before we figure out which pages are in the portal
            var porPageMap = new GlideRecord('sp_page_route_map');
            porPageMap.addNotNullQuery('portals');
            porPageMap.addEncodedQuery('portals=' + portalCheck.sys_id); // updated this query to not include route entries not associated to a portal due to higher chance of unecessary results
            porPageMap.query();
            while (porPageMap.next()) {
                var pages = [];
                pages.push(porPageMap.route_from_page.sys_id); // we need to process the from pages
                pages.push(porPageMap.route_to_page.sys_id); // we also need to process the to pages as well
                pages.sort();

                // we need to make the pages unique
                var cleanPages = new ArrayUtil();
                cleanPages = cleanPages.unique(pages);

                for (var i = 0; i < cleanPages.length; i++) {
                    // now we can process each unique page
                    _getPage(cleanPages[i]);

                    // now we need to check for any Request Filters
                    var reqFil = new GlideRecord('request_filter');
                    reqFil.addQuery('portal_page', cleanPages[i]);
                    reqFil.query();
                    while (reqFil.next()) {
                        lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(reqFil, reqFil.sys_class_name.getDisplayValue(), reqFil.getDisplayValue());
                    }
                }
            }

            // lets check for any categories
            var checkCatCats = new GlideRecord('m2m_sp_portal_catalog');
            checkCatCats.addEncodedQuery('sp_portal=' + portalCheck.sys_id.toString());
            checkCatCats.addQuery('active', 'true');
            checkCatCats.query();
            while (checkCatCats.next()) {
                // we might have more than one catalog associated to the portal, so we need to loop through each of them
                var getCatalog = new GlideRecord('sc_catalog');
                getCatalog.addQuery('sys_id', checkCatCats.sc_catalog.sys_id);
                getCatalog.addQuery('active', 'true');
                getCatalog.query();
                if (getCatalog.next()) {
                    lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(getCatalog, "Service Catalog", "Name");
                }

                // now we need to get the Categories in the Catalog, but we don't need the items in the Catalog (there's a seperate artifact for that)
                var getCatCat = new GlideRecord('sc_category');
                getCatCat.addEncodedQuery('sc_catalog=' + checkCatCats.sc_catalog.sys_id.toString());
                getCatCat.addQuery('active', 'true');
                getCatCat.orderBy('title');
                getCatCat.query();
                while (getCatCat.next()) {
                    lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(getCatCat, checkCatCats.sc_catalog.getDisplayValue() + " - Service Catalog Categories", "Category Name");
                }
            }
            // we also need to check for the KB categories associated to this portal
            var checkKBcats = new GlideRecord('m2m_sp_portal_knowledge_base');
            checkKBcats.addEncodedQuery('sp_portal=' + portalCheck.sys_id.toString());
            checkKBcats.query();
            while (checkKBcats.next()) {
                // we now need to loop through each KB for it's categories
                var getKBcats = new GlideRecord('kb_category');
                getKBcats.addQuery('parent_id', checkKBcats.kb_knowledge_base.sys_id);
                getKBcats.addQuery('active', 'true');
                getKBcats.query();
                while (getKBcats.next()) {
                    lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(getKBcats, checkKBcats.kb_knowledge_base.getDisplayValue() + " - KB Categories", "Category Name");
                }
            }

            // now we need to check the widgets on the defined pages for the portal in question, rather than the ones with the routing
            if (portalCheck.homepage) {
                // if we have a homepage value
                _getPage(portalCheck.homepage.sys_id);
            }
            if (portalCheck.kb_knowledge_page) {
                // if we have a kb page value
                _getPage(portalCheck.kb_knowledge_page.sys_id);
            }
            if (portalCheck.login_page) {
                // if we have a login page value
                _getPage(portalCheck.login_page.sys_id);
            }
            if (portalCheck.notfound_page) {
                // if we have a 404 page value
                _getPage(portalCheck.notfound_page.sys_id);
            }
            if (portalCheck.sc_catalog_page) {
                // if we have a catalog page value
                _getPage(portalCheck.sc_catalog_page.sys_id);
            }
            if (portalCheck.sc_category_page) {
                // if we have a Catalog category home page value
                _getPage(portalCheck.sc_category_page.sys_id);
            }
            if (portalCheck.sp_rectangle_menu) {
                // we need to fetch the widget record
                var getHeaderWid = new GlideRecord('sp_widget');
                getHeaderWid.addQuery('sys_id', portalCheck.sp_rectangle_menu.sp_widget.sys_id);
                getHeaderWid.query();
                if (getHeaderWid.next()) {
                    lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(getHeaderWid, "Header Menu", "Name");
                    lfDocumentContentBuilder.processScript(getHeaderWid.template, "Header Menu", "Body HTML Template");
                    lfDocumentContentBuilder.processScript(getHeaderWid.script, "Header Menu", "Server Script");
                    lfDocumentContentBuilder.processScript(getHeaderWid.client_script, "Header Menu", "Client Controller");
                    lfDocumentContentBuilder.processScript(getHeaderWid.link, "Header Menu", "Link");
                }
                // we also need to check if there's any ${} calls
                _getMessages(portalCheck.sp_rectangle_menu.sp_widget.template.toString(), "Header Menu", "Body HTML Template");
                _getMessages(portalCheck.sp_rectangle_menu.sp_widget.script.toString(), "Header Menu", "Server Script");
                _getMessages(portalCheck.sp_rectangle_menu.sp_widget.client_script.toString(), "Header Menu", "Client Controller");
                _getMessages(portalCheck.sp_rectangle_menu.sp_widget.link.toString(), "Header Menu", "Link");
            }

            // we need to check portal content categories in this way because otherwise it causes duplicates at the page level
            var pluginCheck = new GlidePluginManager();
            var cont = pluginCheck.isActive('sn_cd'); // this is to avoid an error for instances where this plugin is not installed
            if (cont == true) {
                var checkCDcats = new GlideRecord('sn_cd_content_category');
                checkCDcats.addEncodedQuery('active=true');
                checkCDcats.orderBy('order');
                checkCDcats.query();
                while (checkCDcats.next()) {
                    lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(checkCDcats, "Portal Content Experience", "Category");
                }

                // while we're here, lets also get the news from this portal
                var getNews = new GlideRecord('sn_cd_content_news');
                getNews.addEncodedQuery('state=published');
                getNews.query();
                while (getNews.next()) {
                    lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(getNews, "News Item", "Name");
                }

                // while we're here, lets also get the various Portal Content entries
                var getPorCont = new GlideRecord('sn_cd_content_portal');
                getPorCont.addEncodedQuery('content_type=41040dd7237e2300fb0c949e27bf652a^ORcontent_type=a673597a0b4303008cd6e7ae37673a6f^state=published');
                getPorCont.query();
                while (getPorCont.next()) {
                    lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(getPorCont, "Portal Content", "Title");
                }
            }

            // there is a scenario where we may have widgets with no ID's which can't be captured unless we specifically look for them but this will likely include widgets not in the scope of this portal, there is no otherway :(
            var getNoIDWid = new GlideRecord('sp_widget');
            getNoIDWid.addEncodedQuery('id=NULL');
            getNoIDWid.orderBy('name');
            getNoIDWid.query();
            while (getNoIDWid.next()) {
                _getWidgets(getNoIDWid);
            }
        }


        function _getPage(pageSYS) {
            try {
                if (pageSYS) {
                    // we need to check if this page has been already processed
                    if (!pageArr.toString().includes(pageSYS.toString())) {


                        // This might not be needed any more due to the Portal content artifact

                        // we need to pick up the Content Experience elements if this is an ESC Pro portal
                        // need to check if "Content Publishing" is installed first
                        var pluginCheck = new GlidePluginManager();
                        var cont = pluginCheck.isActive('sn_cd'); // this is to avoid an error for instances where this plugin is not installed
                        if (cont == true) {
                            // we only need to know it's installed, so we don't need the full record return

                            // let's now check for what Content is set up
                            var getContExp = new GlideRecord('sn_cd_content_visibility');
                            getContExp.addEncodedQuery('sp_page=' + pageSYS + '^active=true');
                            getContExp.query();
                            while (getContExp.next()) {
                                // we need to process the page directly related to this record
                                if (getContExp.sp_page.sys_id != pageSYS) {
                                    lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(getContExp, "Portal Content Experience", getContExp.title);
                                    _getPage(getContExp.sp_page.sys_id); // we need to send the page associated to the content, which would call this very function again... but we need to know...
                                }

                                // now we need the actual content record
                                var contBase = new GlideRecord('sn_cd_content_portal'); // this should represent the various sub-classes of content
                                contBase.addNotNullQuery('sys_id');
                                contBase.addQuery('sys_id', getContExp.content.sys_id);
                                contBase.query();
                                if (contBase.next()) {
                                    lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(contBase, "Portal Content Experience", getContExp.title);
                                    lfDocumentContentBuilder.processString(contBase.button_text.toString(), "Portal Content Experience", "Button Text");
                                }
                            }
                        }

                        var pageVal = instCheck.sp_column.sp_row.sp_container.sp_page.getDisplayValue().toString();
                        if (!pageVal) {
                            pageVal = "No Page Associated";
                        }

                        // if we have the page sys_id then we need to loop from the instances
                        var instCheck = new GlideRecord('sp_instance');
                        instCheck.addNotNullQuery('sp_column');
                        instCheck.addEncodedQuery('sp_column.sp_row.sp_container.sp_page=' + pageSYS);
                        instCheck.addQuery('active', 'true');
                        instCheck.query();
                        while (instCheck.next()) {
                            lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(instCheck, "Widget Instance - " + instCheck.sys_class_name.getDisplayValue(), "Page - " + pageVal + " > Widget - " + instCheck.sp_widget.getDisplayName());
                            _getWidgets(instCheck, pageSYS);
                        }
                        pageArr.push(pageSYS.toString()); // we need to feed the page just processed into the Page Array
                    }
                }
            } catch (err) {
                //gs.log('Error in _getPage - ' + err.name + ' - ' + err.message);
            }
        }

        function _getWidgets(instCheck, pageSYS) {
            try {
                // we need to check if this widget has already been processed
                var widgetCheck = '';
                if (instCheck) {
                    if (instCheck.sp_widget) {
                        // this covers instances, there are multiple types
                        widgetCheck = instCheck.sp_widget.sys_id.toString();
                    } else {
                        // this covers widget widgets
                        widgetCheck = instCheck.sys_id.toString();
                    }
                    // widget duplication check
                    if (!widgetArr.toString().includes(instCheck.toString())) {
                        widgetArr.push(instCheck.toString()); // now we need to push the widget processed so we can de-dedupe check it for each widget we go through
                    }
                } else {
                    return
                }

                if (widgetCheck == 'cf1a5153cb21020000f8d856634c9c3c') {
                    // this is the ootb carousel widget we now need to check the carousel slides to see if they are related to this portal
                    var checkCarousel = new GlideRecord('sp_carousel_slide');
                    checkCarousel.addEncodedQuery('carousel.sp_widget=' + widgetCheck);
                    checkCarousel.query();
                    while (checkCarousel.next()) {
                        lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(checkCarousel, "Carousel Slides", "Name");
                    }
                }

                if (!widgetArr.toString().includes(widgetCheck.toString()) && pageSYS != '') {
                    var pageVal = '';
                    var pageName = new GlideRecord('sp_page');
                    pageName.addQuery('sys_id', pageSYS);
                    pageName.query();

                    if (!pageName.hasNext()) {
                        if (instCheck.sp_column.sp_row.sp_container.sp_page.getDisplayValue().toString() != '') {
                            // we need to be super defensive
                            pageVal = instCheck.sp_column.sp_row.sp_container.sp_page.getDisplayValue().toString();
                        }
                    }
                    if (pageName.next()) {
                        pageVal = pageName.getDisplayValue();
                    }
                    if (!pageVal) {
                        pageVal = "No Page Associated";
                    }

                    // now we need to check through widgets
                    var widCheck = new GlideRecord('sp_widget');
                    widCheck.addEncodedQuery('sys_id=' + widgetCheck + '^ORsp_widget=' + widgetCheck);
                    widCheck.query();
                    while (widCheck.next()) {
                        lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(widCheck, widCheck.sys_class_name.getDisplayValue(), "Page - " + pageVal + " > Widget - " + widCheck.getDisplayValue());

                        // in some very rare scenarios we also need to check for other titles of a widget instance
                        var getOtherWidInst = new GlideRecord('sp_instance');
                        getOtherWidInst.addEncodedQuery('active=true^title!=NULL^sp_widget=' + widCheck.sys_id);
                        getOtherWidInst.orderBy('order');
                        getOtherWidInst.query();
                        if (getOtherWidInst.next()) {
                            lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(getOtherWidInst, widCheck.sys_class_name.getDisplayValue(), "Page - " + pageVal + " > Widget - " + widCheck.getDisplayValue());
                        }

                        // we need to see if this widget is the GetSupport widget
                        if (widCheck.id == 'ec-get-support') {
                            // this is if it's the new EC "get support" widget, then we need to go and query the sn_ex_sp_get_support for supporting elements
                            var getSupportCheck = new GlideRecord('sn_ex_sp_get_support');
                            getSupportCheck.query();
                            while (getSupportCheck.next()) {
                                // now we need to go to each record
                                var checkSupport = new GlideRecord(getSupportCheck.table.toString()); // this should be holding the table value
                                checkSupport.addQuery('sys_id', getSupportCheck.content.toString()); // this should be holding the sys_id of the linked record
                                checkSupport.query();
                                if (checkSupport.next()) {
                                    lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(checkSupport, "Support items", checkSupport.title.toString());
                                }
                            }
                        }

                        // we need to check the Angular provider as well
                        var checkAng = new GlideRecord('sp_ng_template');
                        checkAng.addQuery('sp_widget', widCheck.sys_id);
                        checkAng.query();
                        while (checkAng.next()) {
                            lfDocumentContentBuilder.processScript(checkAng.template, widCheck.sys_class_name.getDisplayValue(), "Page - " + pageVal + " > Angular Provider - " + checkAng.getDisplayValue());
                            _getMessages(checkAng.template.toString(), widCheck.sys_class_name.getDisplayValue(), "Page - " + pageVal + " > Angular Provider - " + checkAng.getDisplayValue());
                        }

                        // template field
                        lfDocumentContentBuilder.processScript(widCheck.template, widCheck.sys_class_name.getDisplayValue(), "Page - " + pageVal + " > Widget - " + widCheck.getDisplayValue());
                        _getMessages(widCheck.template.toString(), widCheck.sys_class_name.getDisplayValue(), "Page - " + pageVal + " > Widget - " + widCheck.getDisplayValue());

                        // Server Script
                        lfDocumentContentBuilder.processScript(widCheck.script, widCheck.sys_class_name.getDisplayValue(), "Page - " + pageVal + " > Widget - " + widCheck.getDisplayValue());
                        _getMessages(widCheck.script.toString(), widCheck.sys_class_name.getDisplayValue(), "Page - " + pageVal + " > widget - " + widCheck.getDisplayValue());

                        // we need to see if there's widget being loaded by script
                        if (widCheck.script.match(/(\$sp.getWidget)(\(\'|\").*?(\"|\')/gi)) {
                            var widLine = widCheck.script.match(/(\$sp.getWidget)(\(\'|\").*?(\"|\')/gi);

                            // we might have an array
                            var servWid = widLine.toString().split(',');
                            for (var widS = 0; widS < servWid.length; widS++) {
                                if (servWid[widS].toString() != '') {
                                    servWid[widS] = servWid[widS].match(/(?:['"].*?['"])/gi); // we want to strip any quotes to get a clean value
                                    // now we need to remove the quotes if there are any
                                    var cleanWid = servWid[widS].toString().replace(/['"]/gi, ''); // with nothing
                                    // now we need to get the widget before we process it
                                    var getWidID = new GlideRecord('sp_widget');
                                    getWidID.addQuery('id', cleanWid.toString());
                                    getWidID.query();
                                    if (getWidID.next()) {
                                        // process this widget
                                        if (getWidID.sys_id != instCheck.sys_id && (!widgetArr.toString().includes(getWidID.sys_id))) {
                                            _getWidgets(getWidID, instCheck.sp_column.sp_row.sp_container.sp_page.sys_id);
                                        }
                                    }
                                }
                            }
                        }

                        // Client Script
                        lfDocumentContentBuilder.processScript(widCheck.client_script, widCheck.sys_class_name.getDisplayValue(), "Page - " + pageVal + " > Widget - " + widCheck.getDisplayValue());
                        _getMessages(widCheck.client_script.toString(), widCheck.sys_class_name.getDisplayValue(), "Page - " + pageVal + " > Widget - " + widCheck.getDisplayValue());

                        // we need to see if there's widgets being loaded by script
                        if (widCheck.client_script.match(/(spUtil.get\(\'|\").*?(\'|\")/gi)) {
                            var widCLid = widCheck.client_script.match(/(spUtil.get\(\'|\").*?(\'|\")/gi);

                            // we might have an array
                            var clWid = widCLid.toString().split(',');
                            for (var widCL = 0; widCL < clWid.length; widCL++) {
                                if (clWid[widCL].toString() != '') {
                                    // now we need to remove the quote if there are any
                                    var cleanWidCL = clWid[widCL].toString().replace(/'|"/g, ''); // we want to strip any quotes to get a clean value
                                    // now we need to get the widget before we process it
                                    var CLgetWidID = new GlideRecord('sp_widget');
                                    CLgetWidID.addQuery('id', cleanWidCL.toString());
                                    CLgetWidID.query();
                                    if (CLgetWidID.next()) {
                                        // process the widget
                                        if (CLgetWidID.sys_id != instCheck.sys_id && (!widgetArr.toString().includes(CLgetWidID))) {
                                            _getWidgets(CLgetWidID);
                                        }
                                    }
                                }
                            }
                        }
                    }


                    // now we need to check instance specifics
                    // the JSON of the instance
                    var pageArr = [];
                    if (instCheck.widget_parameters != '') {
                        var valueObj = instCheck.widget_parameters.toString();
                        // we need to process the "widget_parameters" incase there's a message call in it
                        if (valueObj != "" || valueObj != " ") {
                            _getMessages(valueObj.toString(), instCheck.sys_class_name.getDisplayValue(), "Page - " + pageVal + " > Widget - " + instCheck.sp_widget.getDisplayValue());
                        }

                        // now we need to see if there's a page in the "widget_parameters"
                        if (valueObj.includes("page") || valueObj.includes("displayValue")) {
                            // we only need to look into the JSON values, if there is a mention of pages otherwise it could be completely unrelated to our needs.
                            var valueCheck = JSON.parse(valueObj, function(key, val) {
                                if (key == 'displayValue') {
                                    // we need this page's sys_id but we need to make sure we don't reprocess the same page we started with
                                    if ((val != pageSYS.id || !valueObj.toString().includes(pageSYS.id)) && val.toString().match(/[a-z0-9]{32}/g)) {
                                        // we don't want to cause a query with a null value and only for a sys_id
                                        var getPageID = new GlideRecord('sp_page');
                                        getPageID.addQuery('id', val.toString());
                                        getPageID.query();
                                        if (!getPageID.hasNext()) {
                                            /*
											// we might not need this after all
											// we might need to factor in a header
                                            if (val.toString() != 'true' && val.toString().match(/(?:^\{)|(^['"].*?['"])/gim) && val.toString() != 'false' && val.toString() != '' && val.toString() != ' ' && val.toString() != "" && val.toString() != 'number') {
                                                lfDocumentContentBuilder.processString(val.toString(), instCheck.sys_class_name.getDisplayValue(), "Page - " + pageVal + " > Widget - " + instCheck.sp_widget.getDisplayValue());
                                            }
											*/
                                        }
                                        if (getPageID.next()) {
                                            pageArr.push(getPageID.sys_id.toString());
                                        }
                                    }
                                    /*
									// we might not need this after all
									else if (val != '[object Object]' && val.toString().match(/(?:^\{)|(^['"].*?['"])/gim) && val.toString() != 'true' && val.toString() != 'false' && val.toString() != '' && val.toString() != ' ' && val.toString() != "" && val.toString() != 'number') {
                                        if (val.toString() != "" || val.toString() != " ") {
                                            lfDocumentContentBuilder.processString(val.toString(), instCheck.sys_class_name.getDisplayValue(), "Page - " + pageVal + " > Widget - " + instCheck.sp_widget.getDisplayValue());
                                        }
                                    }
									*/
                                }
                            });
                        }
                    }
                    var pageIDArr = new ArrayUtil();
                    pageIDArr.unqiue(pageArr);
                    for (var iD = 0; iD < pageIDArr.length; iD++) {
                        _getPage(pageIDArr[iD]);
                    }

                    // now we need to do some specific option_schema checks
                    var optMsg = [];
                    if (instCheck.sp_widget.option_schema != '') {
                        var widString = instCheck.sp_widget.option_schema.toString();
                        if (widString != '') {
                            var widJ = JSON.parse(widString, function(key, value) {
                                // if we send these to the array, we can clear out duplicates
                                if (key == "displayValue" && (value != '' && value != ' ')) {
                                    optMsg.push(value);
                                }
                                if (key == "label" && (value != '' && value != ' ')) {
                                    optMsg.push(value);
                                }
                                if (key == "hint" && (value != '' && value != ' ')) {
                                    optMsg.push(value);
                                }
                                if (key == "section" && (value != '' && value != ' ')) {
                                    optMsg.push(value);
                                }
                                if (key == "default_value" && value.toString().match(/(^\b).*(\b)/gim) && (value.toString() != 'true' && value.toString() != 'false')) {
                                    optMsg.push(value);
                                }
                            });
                        }
                        // we need to strip the option_schema array of any duplicates before we process it
                        optMsg.sort();
                        var optSchArr = new ArrayUtil();
                        optSchArr = optSchArr.unique(optMsg);
                        lfDocumentContentBuilder.processStringArray(optSchArr, instCheck.sys_class_name.getDisplayValue(), "Page - " + pageVal + " > Widget -  " + instCheck.sp_widget.getDisplayValue());
                        optMsg = [];
                    }


                    // we need to check to see if the EC or EC pro is installed first
                    var checkEC = new GlideRecord('sys_package');
                    checkEC.addEncodedQuery('source=sn_ex_sp^ORsource=sn_ex_sp_pro');
                    checkEC.query();
                    if (checkEC.hasNext()) {

                        // for the /esc portal, we also need to loop through any "web applications"
                        if (instCheck.sp_widget.id == 'web_applications' || instCheck.sp_widget.id == "app-launcher") {
                            var webApp = new GlideRecord('sn_ex_sp_pro_web_application');
                            webApp.addNotNullQuery('active');
                            webApp.addEncodedQuery('active=true'); // we only need active items
                            webApp.query();
                            while (webApp.next()) {
                                lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(webApp, "Web Applications", webApp.name);
                            }
                        }

                        // now we need the quick links
                        if (instCheck.sp_widget.id == "quick-links" || instCheck.sp_widget.id == "cd-quick-links" || instCheck.sp_widget.id == "quick_links_on_topic_page") {
                            var quickLinks = new GlideRecord('sn_ex_sp_quick_link');
                            quickLinks.addNotNullQuery('active');
                            quickLinks.addEncodedQuery('active=true'); // these will pick up everything connected via the m2m_connected_content table and any others associated
                            quickLinks.query();
                            while (quickLinks.next()) {
                                lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(quickLinks, "Quick Links", "Name");
                            }
                        }

                        // we need to get the Activity items of task records
                        if (instCheck.sp_widget.id == 'my-items') {
                            var activeItems = new GlideRecord('sn_ex_sp_activity_configuration');
                            activeItems.addNotNullQuery('active');
                            activeItems.addEncodedQuery('active=true');
                            activeItems.query();
                            while (activeItems.next()) {
                                lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(activeItems, "Active Items", "Name");
                            }
                            // we also need to manually add some due time messages for these tasks
                            lfDocumentContentBuilder.processString('overdue', "Active Items", "Name");
                            lfDocumentContentBuilder.processString('overdue {0} day', "Active Items", "Name");
                            lfDocumentContentBuilder.processString('overdue {0} days', "Active Items", "Name");
                            lfDocumentContentBuilder.processString('due today', "Active Items", "Name");
                            lfDocumentContentBuilder.processString('due in {0} days', "Active Items", "Name");
                            lfDocumentContentBuilder.processString('Quick tasks', "Active Items", "Name");
                        }
                    }


                    // now that we've processed this page, let's now process any others we've previously found
                    var cleanPage = new ArrayUtil();
                    cleanPage = cleanPage.unique(pageArr);
                    if (cleanPage.length > 0) {

                        for (var p = 0; p < cleanPage.length; p++) {
                            var pageID = new GlideRecord('sp_page');
                            pageID.addNotNullQuery('id');
                            pageID.addQuery('id', cleanPage[p]);
                            pageID.addQuery('sys_id', '!=', sysId); // just to ensure we're not re-looping
                            pageID.query();
                            if (pageID.next()) {
                                _getPage(pageID.sys_id); // call this function again with the new page
                            }
                        }
                        pageArr = []; // resetting the array
                        cleanPage = []; // resetting the array
                    }
                }
            } catch (err) {
                //gs.log('Error in _getWidgets - ' + err.name + ' - ' + err.message);
            }
        }



        function _getMessages(recField, name, label) {
            try {
                // ${} checks
                var MsgCheck = /\${(?:[^{}]|\{(?:[^{}]|\{[^{}]*\})*\})*}|(?:^\$\{)(.*)(\})/gim; // this is a lighter version but is designed to find ${something} under multiple scenarios
                var pgClean = /(^\$\{|^\{)|(\}$)/gi; // this is for cleaning out the flanking { } in the identified string
                var msgArr = [];

                var fieldCheck = '';
                fieldCheck = recField.toString().match(MsgCheck);

                if (fieldCheck) {
                    var fieldClean = fieldCheck.toString().split(',$');
                    for (var i = 0; i < fieldClean.length; i++) {
                        if (!fieldClean[i].toString().includes('getMessage')) {
                            // this is a defensive check to avoid unnecessary duplicates
                            if (fieldClean[i].toString() != '' && (!fieldClean[i].toString().includes('{{item.label}}'))) {
                                // this is a defensive check because there are some sometimes some MessageAPI calls that contain nothing
                                var cleanStr = fieldClean[i].toString().replace(pgClean, '');
                                if (cleanStr) {
                                    // we need to make sure we're not inadvertantly pushing a blank space
                                    msgArr.push(cleanStr); // send to the array to clean unnecessary duplicates
                                }
                            }
                        } else if (fieldClean[i].toString().includes('getMessage')) {
                            // this might not ever be needed due to use of .processScript() method used on the widget records
                            var msgStringReg = /(['"])(.*)(['"])/gi;
                            var msgString = fieldClean[i].toString().match(msgStringReg);
                            // now we need to move the flanking quotes (single or double)
                            var getMclean = /(\B['"])|(['"]\s)/gi;
                            var cleanGetM = msgString.replace(getMclean, '');
                            msgArr.push(cleanGetM); // send to the array to clean unnecessary duplicates
                        }
                    }
                }

                // getMessage(variable) checks
                var checkMsgReg = /(?:getMessage\()(?!['"])(.*)(?!['"])(\))/gi;
                var callCheck = recField.toString().match(checkMsgReg);

                if (callCheck) {
                    // we need to cleanup the call
                    var callValReg = /([^()]*)/gi;
                    var callValCheck = callCheck.toString().match(callValReg);
                    var varToCheck = '';
                    for (var cvi = 0; cvi < callValCheck.length; cvi++) {
                        if (!callValCheck[cvi].toString().includes('getMessage') && callValCheck[cvi].toString() != '') {
                            varToCheck = callValCheck[cvi].toString();
                        }
                    }

                    // now we need to see if there is a variable of this with a string associated to it
                    // the regex being built is - /(?:(var\s)(something).*(\=)).*(['"].*["';])/gi
                    var checkIfVarReg1 = "(?:(var\s)(";
                    var checkIfVarReg2 = ").*(\=)).*(['";
                    var checkIfVarReg3 = '"].*[';
                    var checkIfVarReg4 = '"';
                    var checkIfVarReg5 = "';])";
                    var buildCheckVarReg = checkIfVarReg1 + varToCheck + checkIfVarReg2 + checkIfVarReg3 + checkIfVarReg4 + checkIfVarReg5;
                    var VarRegChecker = new RegExp(buildCheckVarReg, "gim");
                    var VarRegCheck = recField.toString().match(VarRegChecker);
                    if (VarRegCheck) {
                        // this is for a variable with the string - which should find something like "var something = 'this is my text' or to that effect"
                        // now we need to identify just the pure string, our findings could be in an array
                        for (vriC = 0; vriC < VarRegCheck.length; vriC++) {
                            var VarStrCheck = /(['"].*["'])/gim;
                            var VarRegChecker = VarRegCheck[vri].toString().match(VarStrCheck);
                            if (VarRegChecker) {
                                // now we need to clean the flanking quotes (single or double);
                                var VarRegClStr = /(^['"])|(['"]$)/gim;
                                var VarRegCleanedStr = VarRegChecker.toString().replace(VarRegClStr, '');
                                msgArr.push(VarRegCleanedStr);
                            }
                        }
                    } else {
                        // so if there's no variable is there a function?
                        var checkIfFuncReg = "(?:function ).*?(\()(" + varToCheck + ")(\))";
                        var checkIfFuncChecker = new RegExp(checkIfFuncReg, "gi");
                        var checkIfFunc = recField.toString().match(checkIfFuncChecker);
                        if (checkIfFunc) {
                            // we have a valid find for a function call, now we need to see each call to this function
                            var funcCheckReg = /(?!\bfunction\s+)(?:(\w+)\s*)(?=\()/gi;
                            var nameOfFunc = checkIfFunc.toString().match(funcCheckReg);

                            // now we need to find every time the function is called and pull out it's string
                            // (error)(\(['"]).*(['"])
                            var nameOfFuncCheck = "(" + nameOfFunc + ")(\(['" + '"]).*([' + "'" + '"])';
                            var nameOfFuncChecker = new RegExp(nameOfFuncCheck, "gi");
                            // this will give us -> func = 'something' so we need to clean it up

                            var FuncNameChecker = recField.toString().match(nameOfFuncChecker);
                            // this will make an array of -> functionname('something')

                            if (FuncNameChecker) {
                                for (fni = 0; fni < FuncNameChecker.length; fni++) {
                                    // lets extract the clean text
                                    var cleanFNI = FuncNameChecker[fni].toString().match(/(['"]).*(['"])/gi);
                                    if (cleanFNI) {
                                        // now we need to clean the flanking quotes
                                        var cleanedFNI = cleanFNI.toString().replace(/(^['"])|(['"]$)/gim, '');
                                        if (cleanedFNI) {
                                            msgArr.push(cleanedFNI);
                                        }
                                    }
                                }
                            }
                        }
                    }
                }

                msgArr.sort();
                var cleanMsg = new ArrayUtil();
                cleanMsg = cleanMsg.unique(msgArr);
                if (cleanMsg.length > 0 && cleanMsg != ' ') {
                    lfDocumentContentBuilder.processStringArray(cleanMsg, name, label);
                }

            } catch (err) {
                //gs.log('Error in _getMessages - ' + err.name + ' - ' + err.message + " - " + label);
            }
        }

        return lfDocumentContentBuilder.build();
    },

    /**********
     * Uncomment the saveTranslatedContent function to override the default behavior of saving translations
     * 
     * @Param documentContent LFDocumentContent object
     * @return
     **********/
    /**********
        saveTranslatedContent: function(documentContent) {},
    **********/

    type: 'LF_PortalProcessor'
});

 

 

 

 

 

 

If for what-ever reason, the above doesn't pick something up on a specific page (this might happen when a page is in a forced href for example), I've also created a Page specific Prototype artifact that you can run on that particular page (I called it "LF_PortalPageProcessor", so be mindful of the Script Include name you'll need to make, and the Artifact name for the UI action):

 

 

 

 

var LF_PortalPageProcessor = Class.create();
LF_PortalPageProcessor.prototype = Object.extendsObject(global.LFArtifactProcessorSNC, {
    category: 'localization_framework', // DO NOT REMOVE THIS LINE!

    /**********
     * Extracts the translatable content for the artifact record
     * 
     *  params.tableName The table name of the artifact record
     *  params.sysId The sys_id of the artifact record 
     *  params.language Language into which the artifact has to be translated (Target language)
     * @return LFDocumentContent object
     **********/
    getTranslatableContent: function(params) {
        /**********
         * Use LFDocumentContentBuilder to build the LFDocumentContent object
         * Use the build() to return the LFDocumentContent object
         **********/

        // we need to trawl through this page for all widgets and widget instances
        var tableName = params.tableName;
        var sysId = params.sysId;
        var language = params.language;
        var lfDocumentContentBuilder = new global.LFDocumentContentBuilder("v1", language, sysId, tableName);
        var pageArr = [];

        var spPage = new GlideRecord('sp_page');
        spPage.addQuery('sys_id', sysId); // this is the page from the UIaction
        spPage.query();
        if (spPage.next()) {
            lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(spPage, "Page Name", "Name");
            // now we need to process the page accordingly
            _getPage(spPage.sys_id);
        }


        function _getPage(pageSYS) {
            var msgArr = []; // for later
            // if we have the page sys_id then we need to loop from the instances
            var instCheck = new GlideRecord('sp_instance');
            instCheck.addEncodedQuery('sp_column.sp_row.sp_container.sp_page=' + pageSYS);
            instCheck.addQuery('active', 'true');
            instCheck.query();
            while (instCheck.next()) {
                lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(instCheck, instCheck.sys_class_name.getDisplayValue(), instCheck.sp_widget.getDisplayName());

                // now we need to check through widgets
                var widCheck = new GlideRecord('sp_widget');
                widCheck.addQuery('sys_id', instCheck.sp_widget.sys_id);
                widCheck.query();
                while (widCheck.next()) {
                    lfDocumentContentBuilder.processTranslatableFieldsForSingleRecord(widCheck, widCheck.sys_class_name.getDisplayValue(), widCheck.getDisplayValue());

                    // template field
                    lfDocumentContentBuilder.processScript(widCheck.template, widCheck.sys_class_name.getDisplayValue(), widCheck.getDisplayValue());
                    _getMessages(widCheck.template, widCheck.sys_class_name.getDisplayValue(), widCheck.getDisplayValue());

                    // Server Script
                    lfDocumentContentBuilder.processScript(widCheck.script, widCheck.sys_class_name.getDisplayValue(), widCheck.getDisplayValue());
                    _getMessages(widCheck.script, widCheck.sys_class_name.getDisplayValue(), widCheck.getDisplayValue());

                    // Client Script
                    lfDocumentContentBuilder.processScript(widCheck.client_script, widCheck.sys_class_name.getDisplayValue(), widCheck.getDisplayValue());
                    _getMessages(widCheck.client_script, widCheck.sys_class_name.getDisplayValue(), widCheck.getDisplayValue());
                }

                // now we need to check the JSON of the instance
                if (instCheck.widget_parameters != '') {
                    var valueObj = instCheck.widget_parameters.toString();
                    if (valueObj.includes("page")) {
                        // we only need to look into the JSON values, if there is a mention of pages otherwise it could be completely unrelated to our needs.
                        valueCheck = JSON.parse(valueObj, function(key, val) {
                            if (key == 'displayValue') {
                                // we need this page's sys_id but we need to make sure we don't reprocess the same page we started with
                                if (val != pageSYS.id || !valueObj.toString().includes(pageSYS.id) || val != '') {
                                    pageArr.push(val);
                                }
                            }
                        });
                    }
                }
                if (instCheck.sp_widget.option_schema != '') {
                    var widString = instCheck.sp_widget.option_schema.toString();
                    var widJ = JSON.parse(widString, function(key, value) {
                        // if we send these to the array, we can clear out duplicates
                        if (key == "displayValue") {
                            msgArr.push(value);
                        }
                        if (key == "label") {
                            msgArr.push(value);
                        }
                        if (key == "hint") {
                            msgArr.push(value);
                        }
                        if (key == "section") {
                            msgArr.push(value);
                        }
                        if (key == "default_value") {
                            msgArr.push(value);
                        }
                    });
                }
                // message array duplicate check
                msgArr.sort();
                var arrayUtil = new ArrayUtil();
                arrayUtil = arrayUtil.unique(msgArr);
                lfDocumentContentBuilder.processStringArray(arrayUtil, instCheck.sys_class_name.getDisplayValue(), instCheck.sp_widget.getDisplayValue());
                msgArr = []; // we need to reset the array
            }


            function _getMessages(recField, name, label) {
				// ${} checks
                var MsgCheck = /\${(?:[^{}]|\{(?:[^{}]|\{[^{}]*\})*\})*}|(?:^\$\{)(.*)(\})/gim; // this is a lighter version but is designed to find ${something} under multiple scenarios
                var pgClean = /(^\$\{|^\{)|(\}$)/gi; // this is for cleaning out the flanking { } in the identified string
                var msgArr = [];

                var fieldCheck = '';
                fieldCheck = recField.toString().match(MsgCheck);

                if (fieldCheck) {
                    var fieldClean = fieldCheck.toString().split(',$');
                    for (var i = 0; i < fieldClean.length; i++) {
                        if (!fieldClean[i].toString().includes('getMessage')) {
                            // this is a defensive check to avoid unnecessary duplicates
                            if (fieldClean[i].toString() != '' && (!fieldClean[i].toString().includes('{{item.label}}'))) {
                                // this is a defensive check because there are some sometimes some MessageAPI calls that contain nothing
                                var cleanStr = fieldClean[i].toString().replace(pgClean, '');
                                if (cleanStr != '') {
                                    // we need to make sure we're not inadvertantly pushing a blank space
                                    msgArr.push(cleanStr); // send to the array to clean unnecessary duplicates
                                }
                            }
                        } else if (fieldClean[i].toString().includes('getMessage')) {
                            // this might not ever be needed due to use of .processScript() method used on the widget records
                            var msgStringReg = /(['"])(.*)(['"])/gi;
                            var msgString = fieldClean[i].toString().match(msgStringReg);
                            // now we need to move the flanking quotes (single or double)
                            var getMclean = /(\B['"])|(['"]\s)/gi;
                            var cleanGetM = msgString.replace(getMclean, '');
                            msgArr.push(cleanGetM); // send to the array to clean unnecessary duplicates
                        }
                    }
                }

                // getMessage(variable) checks
                var checkMsgReg = /(?:getMessage\()(?!['"])(.*)(?!['"])(\))/gi;
                var callCheck = recField.toString().match(checkMsgReg);

                if (callCheck) {
                    // we need to cleanup the call
                    var callValReg = /([^()]*)/gi;
                    var callValCheck = callCheck.toString().match(callValReg);
                    var varToCheck = '';
                    for (var cvi = 0; cvi < callValCheck.length; cvi++) {
                        if (!callValCheck[cvi].toString().includes('getMessage') && callValCheck[cvi].toString() != '') {
                            varToCheck = callValCheck[cvi].toString();
                        }
                    }

                    // now we need to see if there is a variable of this with a string associated to it
                    // the regex being built is - /(?:(var\s)(something).*(\=)).*(['"].*["';])/gi
                    var checkIfVarReg1 = "(?:(var\s)(";
                    var checkIfVarReg2 = ").*(\=)).*(['";
                    var checkIfVarReg3 = '"].*[';
                    var checkIfVarReg4 = '"';
                    var checkIfVarReg5 = "';])";
                    var buildCheckVarReg = checkIfVarReg1 + varToCheck + checkIfVarReg2 + checkIfVarReg3 + checkIfVarReg4 + checkIfVarReg5;
                    var VarRegChecker = new RegExp(buildCheckVarReg, "gim");
                    var VarRegCheck = recField.toString().match(VarRegChecker);
                    if (VarRegCheck) {
                        // this is for a variable with the string - which should find something like "var something = 'this is my text' or to that effect"
                        // now we need to identify just the pure string, our findings could be in an array
                        for (vriC = 0; vriC < VarRegCheck.length; vriC++) {
                            var VarStrCheck = /(['"].*["'])/gim;
                            var VarRegChecker = VarRegCheck[vri].toString().match(VarStrCheck);
                            if (VarRegChecker) {
                                // now we need to clean the flanking quotes (single or double);
                                var VarRegClStr = /(^['"])|(['"]$)/gim;
                                var VarRegCleanedStr = VarRegChecker.toString().replace(VarRegClStr, '');
                                msgArr.push(VarRegCleanedStr);
                            }
                        }
                    } else {
                        // so if there's no variable is there a function?
                        var checkIfFuncReg = "(?:function ).*?(\()(" + varToCheck + ")(\))";
                        var checkIfFuncChecker = new RegExp(checkIfFuncReg, "gi");
                        var checkIfFunc = recField.toString().match(checkIfFuncChecker);
                        if (checkIfFunc) {
                            // we have a valid find for a function call, now we need to see each call to this function
                            var funcCheckReg = /(?!\bfunction\s+)(?:(\w+)\s*)(?=\()/gi;
                            var nameOfFunc = checkIfFunc.toString().match(funcCheckReg);

                            // now we need to find every time the function is called and pull out it's string
                            // (error)(\(['"]).*(['"])
                            var nameOfFuncCheck = "(" + nameOfFunc + ")(\\(['" + '"]).*([' + "'" + '"])';
                            var nameOfFuncChecker = new RegExp(nameOfFuncCheck, "gi");
                            // this will give us -> func = 'something' so we need to clean it up

                            var FuncNameChecker = recField.toString().match(nameOfFuncChecker);
                            // this will make an array of -> functionname('something')

                            for (fni = 0; fni < FuncNameChecker.length; fni++) {
                                // lets extract the clean text
                                var cleanFNI = FuncNameChecker[fni].toString().match(/(['"]).*(['"])/gi);
                                if (cleanFNI) {
                                    // now we need to clean the flanking quotes
                                    var cleanedFNI = cleanFNI.toString().replace(/(^['"])|(['"]$)/gim, '');
                                    if (cleanedFNI) {
                                        msgArr.push(cleanedFNI);
                                    }
                                }
                            }
                        }
                    }
                }

                msgArr.sort();
                var arrayUtil = new ArrayUtil();
                arrayUtil = arrayUtil.unique(msgArr);
                if (arrayUtil.length > 0 && arrayUtil != ' ') {
                    lfDocumentContentBuilder.processStringArray(arrayUtil, name, label);
                }

            }
        }

        // now that we've processed this page, let's now process any others we've previously found
        var cleanPage = new ArrayUtil();
        cleanPage = cleanPage.unique(pageArr);
        if (cleanPage.length > 0) {
            for (var p = 0; p < cleanPage.length; p++) {
                var pageID = new GlideRecord('sp_page');
                pageID.addQuery('id', cleanPage[p]);
                pageID.addQuery('sys_id', '!=', sysId); // just to ensure we're not re-looping
                pageID.query();
                if (pageID.next()) {
                    _getPage(pageID.sys_id); // call this function again with the new page
                }
            }
            pageArr = []; // resetting the array
            cleanPage = []; // resetting the array
        }
        return lfDocumentContentBuilder.build();
    },

    /**********
     * Uncomment the saveTranslatedContent function to override the default behavior of saving translations
     * 
     *  documentContent LFDocumentContent object
     * @return
     **********/
    /**********
        saveTranslatedContent: function(documentContent) {},
    **********/

    type: 'LF_PortalPageProcessor'
});

 

 

 

 

 

 

For those who want to learn, have a look at some of my comments in the code to get an idea of my reasoning for certain concepts and the way I structured my queries. Which also serves as a good example of Coding Standards and ensuring you comment your code.

With regards to the UIaction (for the portal level artifact) it should be defined on the [sp_portal] table, and should look like this:
find_real_file.png

 

* NOTE - in the "condition" of the UI Action, we need to call the "internal name" of the artifact, your's may be different to my example, ultimately if nothing happens when you click the UI Action, it will be because the call isn't recognising your artifact's "internal name".

 

 

When it all comes together, and you're able to request a translation of the portal, the comparison UI will break out the components into the following sub-sections (there could be more or less, depending on your portal):
find_real_file.png 

 

Where-by you will be able to interact with the translations (just like you do with Catalog Items) like this:
find_real_file.png

 

 

Summary

What have we learned? Well, providing we understand how the 5 tables work and providing we understand the hierarchy of the "thing" we want to translate and therefore localize, it's very possible to make the method of actioning simpler. This method does not require a single spreadsheet or very complicated exports and imports.

Over time I can imagine some very clever Artifacts being made out there, but at least the 3 I've provided here will serve as a good starting point.

 

So, when it's all set-up you should be able to achieve this:


 

Feel free to tweak, modify, optimize this one as I only wrote it in a very short period of time so as to show the art of the possible, as well as help explain to others how to demystify the seemingly complex world of Localizing a Portal.

 

 

As always, if you liked this, please consider subscribing and like and share if you want to see more as it always helps

 

 

 

Comments
Brent Caudill
Tera Contributor

Hi @Alex Coope - SN  - 

 

I checked the 'Application Restricted Caller Access' table and didn't find anything in there.

 

I did notice that it was trying to query the HR tables, but we don't have any of the HR apps installed, and those tables don't exist in our environment. There are no HR related scopes available either, and there shouldn't be any delegated admin permissions setup.

I tried to run the request translations again in Dev and also in Test and am not see the same warning message with all of the query errors, but am seeing the following query error consistently:

Invalid query detected, please check logs for details [Unknown field sp_widget in table sp_widget]

Brent Caudill
Tera Contributor

Hi @Alex Coope - SN ,

 

Wanted to share one more bit of information in case it triggers anything for you. Otherwise, I'll keep digging on my end to find a workaround. I really appreciate the suggestions/assistance you've provide.

 

I ran the script tracer while attempting to request translations and it is throwing two errors for line 155. Not sure if it's a coincidence that it is the same line in both files, or it isn't reporting correctly. 

 

File Name: LF_PortalProcessor
Line no. 155 | if (footerMenuItem.next())

Error: Unexpected token: u

 

File Name: JSON
Line no. 155 | return NativeJSON.parse.apply(NativeJSON, arguments);
Error: Unexpected token: u

 

Transaction:
request-parameters: api:api
url: http://<instance>.service-now.com/api/sn_lf/localization_framework/requestTranslations
site: api
name: #1019694 /api/sn_lf/localization_framework/requestTranslations
startTime: 2024-10-03 07:48:59 AM
page: api/sn_lf/localization_framework/requestTranslations

LF_ScriptTracer.png

Andrii Valiiev
Tera Contributor

@Alex Coope - SN Alex, thanks a lot for this post - it helps a lot in Full Portal translation.

 

Yes, I had to rewrite many things, add more specific dependencies, add multiple conditions related to the latest release of Employee Center, add translations of related configuration tables for To-Dos, Request Filters, Activities etc.-etc., but, based on your extensive Portal Artefact MVP, now I have a one-click translation tool for the full EC / EC Pro Portal ðŸ”¥ðŸ”¥ðŸ”¥ Thanks a lot!

 

Nevertheless, the Localization Framework toolkit seems to be not yet fully ready. Hope you and your team have time to fix/improve Localization Framework and you are open for suggestions.

 

I've faced several things that were making the process slower, sometimes more manual than anticipated, sometimes requiring extra data manipulations. There was nothing that was not possible to be solved (having time, hummer and a lot of enthusiasm), but I personally see the following items as good improvements (sure, if you and your team have capacity to address them):

 

1) "Unique ID for each row in exported file"

Translation records are exported to CSV with [3] columns: (a) composed title, (b) English text, (c) target language text -- but the first column (which is a "composed title") is not truly unique; so, if I work with the same LFTASK - all is good, but if I need to re-generate my translations, then the CSV file with provided translations based on the original LFTASK might not match - and I've had a lot of fields that were incorrectly mapped (mixed). This has happened because while we were waiting for the translations to be provided by the client's translation partner agency, a clone down to ServiceNow instance has happened where the original LFTASK was generated, so the LFTASK was lost and we had to re-generate our translations. Sure, it had to be properly organized etc., but it would be much better if exported CSV file contains unique IDs (at least where possible), e.g. Sys ID of the record if the record already exists, or Sys ID of the Document if the record has a Document ID reference, or some kind of MD5SUM (or anything else) that can be the same even if you re-generate your translations set.

 

2) "Add non-scoped translated data to [LF: Translations] Update Set as well"
"Translated Text" [sys_translated_text] records are "data", not scoped, have no binding to an Application [sys_scope]. This might be the reason that they were not added to [LF: Translations] Update Set automatically, though definitely were expected to be added. In my case, while processing translations for the full EC Pro Portal to [6] languages, I had about [500] records added to [LF: Translations] Update Set, while not having any single "Translated Text" [sys_translated_text] record added to Update Set. And I had [4000+] translated records that were added to "Translated Text" [sys_translated_text] but ignored by [LF: Translations] Update Set.
 
3) "Proper calculation of Translation Conflicts"
While working with translations on UI Page with Compare Translations layout, translation conflicts are shown as "1 of 1", no matter how many conflicts are actually there. Total number of collected records is properly calculated, as well as a total number of already translated ones among them. But the number of conflicts is always shown as "1 of 1".
 
4) "Show only missing translations"
While working with translations on UI Page with Compare Translations layout, would be nice to have an action link something like "show only missing translations". Right now there is an action link that collapses all groups with everything translated and keeps open those groups that have some translations missing - this is already very helpful, especially when translating a lot of things all at once, like the Full Portal. But still it would be better to have that kind of "show only missing translations" action link, so that all already translated records are hidden, not just collapsed, and the user can see (and ideally export/import) only those missing translations. This "only missing translations" sub-list can be opened in a new tab, if necessary, if it makes the implementation easier. It would also give a much easier possibility to provide incremental sub-sets of missing translations. For example, right now my Full Portal translation for the client has about [2000] records per language, while only [35] records have missing translations. Next month they might add some Quick Links, modify something in Taxonomy, and fix some texts in Rich Text under Content Delivery & Publishing - and, by default, I will have to provide their partner translation agency with the whole [2000+] records CSVs, instead of just about [50] incremental records. And if I send them only those [50] incremental records (manually cut from the full CSV file), then it will require manual export file preparation and manual upload of translated values back, which we definitely want to avoid with Localization Framework.
 
5) "Show only translations with conflicts"
While working with translations on UI Page with Compare Translations layout, would be nice to have an action link something like "show only translations with conflicts". Right now there is a "translation conflicts" warning shown on top of the page, and then each conflict record has a warning text below its translated value, so I can search through the page for that warning message and find all those conflict records - this is already very helpful, especially when translating a lot of things all at once, like the Full Portal. But still it would be better to have that kind of "show only translations with conflicts" action link, so that all records without conflicts are hidden, not just collapsed, and the user can see (and ideally export/import) only those conflict translations. This "only translations with conflicts" sub-list can be opened in a new tab, if necessary, if it makes the implementation easier. Also this sub-list can directly have [4] columns, instead of [3], showing (a) composed title, (b) English text, (c) original translation, (d) translator's text; ideally with inline possibility (radio button) to select (c) or (d), and then "Save" altogether - this will help a lot to resolve the conflicts, as right now you need to click on each conflict individually, "Save" individually, and you are auto-scrolled to the top of the page, so you need to search for the next conflict after that.
 
6) "Solve all conflicts"
While working with translations on UI Page with Compare Translations layout, would be nice to have an action button something like "Solve all conflicts". A modal window (or a new tab) can be opened showing choices: "apply all changes from Translator" vs. "revert all to Original values". Ideally this modal window (or a new tab) should show the list of all conflicts, having [4] columns: (a) composed title, (b) English text, (c) original translation, (d) translator's text - so that it is possible to review the whole list of conflicts one more time. The reason for this is that the person who is uploading the translations (especially for CSV export/import use case with external translation partner agency) is most probably not a translator but an admin or a manager, and this person has no way (no knowledge, no expertise) to actually review the conflicts - this person is ready to resolve all conflicts towards Translator's texts or revert to originals, but not really verify anything. Instead, right now this person has to manually click on each conflict and then "Save", again and again, and it might take a while, and increase the human errors. And also Translation UI Page does not allow to Publish translations unless all conflicts are solved - so there's no other way than manually click, click, click.
 
7) "Autosave"
While working with translations on UI Page with Compare Translations layout, especially when the list is really huge, would be great to add something like an Autosave each XX seconds/minutes. If you have a big translation document (like in my case with Full Portal translation), you might end up with session timeout and all your changes will be lost. As a workaround (after it happened to me), I was pressing "Save as Draft" button from time to time. But it will be much better if there is an option (or a setting) to define an Autosave, which will save your changes (so that you do not lose all of them) and also keep your session alive - perform same "Save as Draft" but automatically.
 
😎 "Consider multiple scopes, not only Global"
All updates are collected under "Global" scope [LF: Translations] Update Set, while there could be the records that belong to different scopes, like "Employee Center Core", "Human Resources: Core" etc. Right now, they all are collected under "Global" scope [LF: Translations] Update Set, which is good, but committing this mixed Update Set to upper env. fails say something like "Cannot commit Update Set 'LF: Translations' because: Update scope id '3d1da2705b021200a4656ede91f91ab6' is different than update set scope id 'global'. Resolve the problem before committing.". So, though all the changes were collected (=good), they have to be collected in their own dedicated Update Sets, one Update Set per each scope, otherwise we are good from translation point of view, but then the Admin will have some painful time while trying to apply these changes to upper instances. I ended up manually splitting single "Global" scope [LF: Translations] Update Set to sub-sets in DEV env., collecting all of them under a Batch Set, and then moving to upper instances. But would really expect that the Localization Framework can do this split (creation of dedicated Update Sets) automatically and group them under [LF: Translations] Update Set as a Batch Set. Use Case: changes in other scopes happen when client wants to modify some of already existing translations (that were delivered under specific scopes, like "Employee Center Core", or "Human Resources: Core" etc.) due to client's communication strategy or specific tone of voice, and they figure out correct (from pure translation point of view) but improper (from internal communication tone of voice point of view) translations.
 
9) "Collect ONLY translations, skip translatable records"
For some reason, [LF: Translations] Update Set took also (maybe not all, but at least some of them) original records that were translated: Menu Items, Taxonomy Topics, Widget Instances etc. Those records should not be considered by [LF: Translations] Update Set because those records are usually scope-dependent, were modified earlier during implementation phase of the project (while translation happens in later, pre-PROD, phase) and had their specific delivery strategy - only translations should be grabbed by [LF: Translations] Update Set, means only those famous [5] tables where translations can be found, nothing more.
 
Yes, I know - long read, a lot of text. But I've tried to explain each item and why it is important.
One more time: none of these items are critical, I was able to proceed through all of them, but it took me much of manual actions, frustration and time-time-time. While I believe that some (or even most) of these items can be easily implemented and this will make the life of people who use Localization Framework much easier and happier.
 
Thanks for the Localization Framework overall.
Thanks for Portal Translation Artefact MVP.
And thanks in advance for any improvements if you and your team have time to work on them.
Alex Coope - SN
ServiceNow Employee
ServiceNow Employee

Hi @Andrii Valiiev,

Some interesting points there. I'll try and cover each point in as short a way as possible:
1) "Unique ID for each row in exported file"

  • Best practice would be to note use CSV with translations because due to the very nature of how CSV files are structured, can pose a real headache with translations if the strings contain a string (CSV = Comma Separated Value). We only have the export to CSV feature as a last resort backup. XLIFF (which is an XML standard used in the translation world) is the better option for any file exports. In terms of the uniqueness of the strings, it depends on the type of translation string source type. For example, an entry to [sys_translated_text] or a TRT would be fairly easy to have a string UUID, just like an entry to [sys_ui_message] or an MSG entry but the others not so much, unless we included multiple attributes about that string which then opens a rabbit hole of other things.

2) "Add non-scoped translated data to [LF: Translations] Update Set as well"

  • TRT entries are a bit of an oddity in that they actually aren't just data. They are binded to the original source record from where the "translated_html" or "translated_text" field originates from. This is how the "XMLSerializer" works. If you haven't read my blog post on how the 5 translation tables work, you mind find this interesting. So, there are actually some technical limitations on whether non-scoped / out of scope entries can be in the update-set. Going forwards (Washington+) the Localization Framework should be generating an Update-set per scope for that LFtask, meaning that you should see multiples with the same name in the various scopes because of this specific reason.

3) "Proper calculation of Translation Conflicts"

  • Indeed, and I agree the UX could be cleaner to represent that there could be more than one conflict. The reason for the "1 of 1" is because each conflict is calculated at the individual level. When a TMS is used, it's not uncommon for a translator to propose multiple alternative translations, so you could actually have a "3 of 1" scenario. 

4) "Show only missing translations"

  • I agree. The task should always contain all translations because there's very real scenarios where the source string has changed and the translation may now be wrong and so we should always assume somebody would want to change a pre-existing translation. Even for scenarios where there has been feedback and the objective is to just update the translation. However, yes I do agree it would be nice to filter down to just those that are missing.

5) "Show only translations with conflicts"

  • Same as I above, I agree.

6) "Solve all conflicts"

  • This one is a bit tricky, because like I mentioned before, in the use-case of using a TMS, a translator might propose multiple alternatives. So offering a "solve all" could inadvertently pick the wrong proposal. But there are still other things that could be done here.

7) "Autosave"

  • I agree and am acutely aware of this one.

😎 "Consider multiple scopes, not only Global"

  • As mentioned above, this isn't technically feasible due to how Application Scopes work or how the update-set migrator works either (aka this is an architectural and security restriction at the platform level which is as expected). However, this is why we have the "Hub and Spoke" system in the Localization Framework, which might be worth having a look at.

9) "Collect ONLY translations, skip translatable records"

  • As mentioned previously, this is very likely down to the the translation type and how those specific types are held in the platform at the architectural level - worth reading the blog post I mentioned above about how the 5 tables work and how their stored in update-sets. So this isn't a limitation in the Localization Framework more about how certain translatable field types are stored in XML.

 

So, thanks for the feedback, it's greatly appreciated and certainly some good ideas. Thanks for taking the time to write such a detailed set of notes,

Many thanks,
kind regards

Andrii Valiiev
Tera Contributor

@Alex Coope - SN thank you for the detailed response.

Seems like I've missed some things, so will check those articles mentioned by you.

MURALIR20469232
Tera Guru

muralimural_0-1735291572708.png

Even I checked the internal name and it was the valid internal name, I am getting this error. When I checked in system log, I found that some error messages.  Please guide me how to proceed further.

Error messages:

 

MURALIR20469232_0-1735292342687.png

 

brumiou
Mega Guru

@Alex Coope - SN Hi Alex, thank you for this brilliant post , that will help me a lot.

I have to translate everything not because we want to translate the instance but just because we want to have a virtual agent in the language of the user. By chance, don't you know any way to have the virtual agent in the language of the user but to let the instance in English? The issue is that they are both based on the preferred_language field on the user profile.

thanks a lot!

 

Xavier

Alex Coope - SN
ServiceNow Employee
ServiceNow Employee

Hi @brumiou,

Our best practice is to think about what the experience would be like for the end-user (especially if they don't understand English as well as your or I), therefore the Portal's UI and each touch point (VA being one) should be considered. So, all of the Catalog Items, KB articles etc. Otherwise, how would that user be able to interact and consume the services available to them?

In terms of the mechanics of user session language, yes it is via the user preference and the value in "preferred_language" which should always be the same (there is logic ootb that keeps them the same). If you try to change this it would cause a damaging and unnecessary technical debt that would prove very problematic and we strongly advise you don't make any changes as it will impact a lot of different features / functionalities across the platform.

For translating VA topics, the LF artifact info (because it's an ootb provided artifact) is available here,

Many thanks,
Kind regards

Philippe Luickx
Tera Contributor
sn_cd_content_category is a table that does not exist in Washington at least, I assume it's part of content publishing framework that is deprecated.
Line 451 queries this table and it's throwing an error for me.
@Brent Caudill I was having the same error messages as you, just commenting out those lines did it for me.
Philippe Luickx
Tera Contributor

@brumiouthis can be done by creating a custom topic.

Show a list of languages.

Then set a context variable (vaVar). Can't remember exactly which one it was, but there should be an OOB language detection topic where you can analyze the script to see what variable name it should be.

 

I agree though that this is a mixed language experience and not recommended. With this great script you can just offer the whole portal in any language with one click, so why not do that?

Sudipta Goswami
Tera Contributor

Thank you @Alex Coope - SN for this article.

 

However, when I try to translate, it throws an error - "Failed to request translations."

I investigated to see why this was coming and mimicked the invoking script. Below is the script that I am running from the "global" scope in the background. 

var data = {
	artifactConfigInternalName: "sp_portal",
	requestType: "form",
	sourceLanguage: "en",
	sysIDs: ["0c3936971b78a55092ad99798b4bcb4f"],
	tableName: "sp_portal",
	targetLanguages: ["id"]
};



var artifactConfigInternalName = data.artifactConfigInternalName;
var itemIds = data.sysIDs;
var targetLanguages = data.targetLanguages;
var sourceLanguage = data.sourceLanguage;
var requestType = data.requestType;

var additionalInfo = {
            "sourceLanguage": sourceLanguage,
            "sourceOfRequest": requestType
        };

var lfTranslations = new LFTranslations();
var result = lfTranslations.requestTranslations(artifactConfigInternalName, itemIds, targetLanguages, additionalInfo);
gs.info(JSON.stringify(result));

 

The Output of the above script is like this - 

 

Attempted script access to inaccessible member denied - java.lang.NullPointerException:getMessage:()Ljava/lang/String;
sys_id="064d8e45c3a31210a36efcbc7a0131fc" sequence="194d114ca410000001" level="2" source="com.glide.ui.ServletErrorListener" message="*** Script: Localization Framework: Error occurred in LFRequestedItemUtils. Error Occurred while creating LF Requested Item for sp_portalwith sys_id 0c3936971b78a55092ad99798b4bcb4ffor language id. JavaException: java.lang.SecurityException: Illegal access to getter method getMessage in class java.lang.NullPointerException: no thrown error" sys_class_name="syslog" context_map="{"_page_name":"sys.scripts.do","_system_id":"app12.dus2.service-now.com.instancename060","_logged_in_user":"sudipta.goswami@xx11.com","_script_record":"05a08653c7912010XXXXXXc427c260a1","_scope":"global","_user":"sudipta.goswami@xx11.com","_is_impersonating":"false","_session_id":"FD03C64DC36XXXXXX6EFCBC7A01315B","_txid":"4a4XXXX5c3a3","_script_table":"sys_script_include"}"
*** Script: Localization Framework: Error occurred in LFRequestedItemUtils. Error Occurred while creating LF Requested Item for sp_portalwith sys_id 0c3936971b78a55092ad99798b4bcb4ffor language id. JavaException: java.lang.SecurityException: Illegal access to getter method getMessage in class java.lang.NullPointerException: no thrown error
*** Script: {"status":"Error","details":[{"sysId":"0c3936971b78a55092ad99798b4bcb4f","targetLanguage":"id","isError":true,"code":40066,"message":"Failed to create requested item for sys id 0c3936971b78a55092ad99798b4bcb4f, source language en, and target language id."}]}

 

It seems, one of the method is failing due to restericted access, as a result the Localization Task record is not getting generated, and "Request Translation" is failing with the above message on the form. I am not able to figure out what, where and how to fix this.

 

NOTE - I created the Artifcat and the Processor script (your previous version) in Global application. 

 

Please review and let me know if you have any suggestion on this. 

 

Thank you again for your help!

Alex Coope - SN
ServiceNow Employee
ServiceNow Employee

Hi @Sudipta Goswami

it looks like you're trying to request for a language of "id" (which would be Bahasa Indonesian), do you have a record in [sys_language] with that value in it's id field? If so, is it active and I assume you have that language associated to the artifact setting?

As the Artifact and Processor script were created in the "global" scope, I'm assuming you're requesting from within the global scope also?

Each of the main functions has a try/catch statement, where the gs.log has been commented out. So, if you need to debug further and you are doing this on a sub-prod you can try uncommenting out those gs.log's and requesting again to generate them. Just remember to comment them once sorted,

Many thanks,
kind regards 

Sudipta Goswami
Tera Contributor

Hello @Alex Coope - SN 

 

Thank you for getting back to me. 

- The Bahasa Indonesia is Active in our instance, and for this we have the localization settings associated already. 

- The BG Script I was running from Global scope only. 

 

Let me add some logs to your processor script to see if it reveals anything.

 

Thank you!

 

Sudipta Goswami
Tera Contributor

Hello @Alex Coope - SN 

 

Finally, I was able to get rid of the error message. 

In the processor script, where you are checking the "sys_package" table (inside _getPage()) and then checking the records from "sn_cd_content_visibility", it was failing there. 

 

We are on Xanadu Patch 5, and In our instance, the table "sn_cd_content_visibility" does not exist. I am not sure what is the purpose of this table and did not get time to investigate more. But after commenting that particular code snippet, I see the Localization Task is getting generated correctly. I will spend some more time tomorrow on this.

 

Thank you and have a good rest of the day!

 

- Sudipta

 

Alex Coope - SN
ServiceNow Employee
ServiceNow Employee

Hi @Sudipta Goswami,

Ah, that's for when "Portal Content" experiences is used (I believe it's now deprecated) so I might update the artifact to have a defensive check for the app scope rather than directly query for the table first. Good find - thanks,

Many thanks,
kind regards 

dk10
Tera Contributor

@Alex Coope - SN  Thanks for this portal artefact.

With this portal artefact we were able to create LFTASK and translation being retrieved successfully from TMS vendor however the associated LFTASK is getting stuck in under review state in servicenow.
Is there any limitations with number of source strings which are retrieved and published in servicenow?


Alex Coope - SN
ServiceNow Employee
ServiceNow Employee

Hi @dk10,

There shouldn't be. It could just be taking a (very) long time. In testing, we have seen some TMS' process 20,000+ strings via this artifact in hours and in an extreme scenario over a day. 

 

Have you been able to confirm the status on the TMS side, and whether the flow running the LFtask hasn't timed out?

 

Many thanks,
Kind regards

dk10
Tera Contributor

@Alex Coope - SN 
Checked with TMS side and translations are already processed and completed in TMS application. We waited for day and two but still the LFTASK is in under review state. When checked at LFTASK fulfilment flow, state is changed from in progress to PRESUMED_INTERRUPTED.

Sudipta Goswami
Tera Contributor

Hello @dk10 @Alex Coope - SN 

 

I have been working on a Localization project and tried translating one of our portals which generated different types of 45K+ records for one language. For us, it took almost an hour for the translations to save the records in different SN message tables and then "Close complete" the LTask and LRitm from the "Under Review" state. I was in the Dev instance and X Patch 5.

 

Because of the time it takes to process those many records, I have been analyzing the OoB UI Page, UI Macro. REST WS /api/sn_lf/localization_framework/publishTranslation API, LFRequestedItemUtils, LFTaskUtils, LFReadSaveScriptUtils etc. Script Includes but did not find any flow (flow designer) associated with the LFTask nor the LFRitm.

 

Now I am wondering if I am missing anything related to flow as you said you got LFTask fulfilment workflow. Could you please let me know what is the flow name you found, and what workflow option you chose in your  Localization settings?

 

Thank you in advance for your help.

Alex Coope - SN
ServiceNow Employee
ServiceNow Employee

Hi @dk10@Sudipta Goswami,

The flow in question is called "Localization Task Fulfilment", you should be able to find it's status in the flow execution table,

Many thanks,
Kind regards  

shivani_verma
Tera Contributor

Hello Alex,

 

Thank you for your detailed information on translations. I followed all your steps to implement the same, but I cannot find Request Translation UI Action anywhere on the portal. I am wondering if you have shown that part in your session. Could you please help here, why I am not able to see the same?

Also, PFA the screenshot of my UI Action (hr_portal is the internal name of my artifact).

 

Thank you in advance!image.jpg

shivani_verma
Tera Contributor

Hello Alex,

 

Thank you for your detailed information on translations. I followed all your steps to implement the same, but I cannot find Request Translation UI Action anywhere on the portal. I am wondering if you have shown that part in your session. Could you please help here, why I am not able to see the same?

Also, PFA the screenshot of my UI Action (hr_portal is the internal name of my artifact).

 

Thank you in advance!image.jpg

Alex Coope - SN
ServiceNow Employee
ServiceNow Employee

Just for thread consistency, I answered the question about the UI action in this thread

jain0811
Tera Explorer

Hi @Alex Coope - SN ,

I followed all the steps successfully, but when I clicked on Request Translation, I encountered the error shown below. Do you have any idea what might be causing this? 

jain0811_0-1754485144415.png

when checked the system logs getting these errors.

jain0811_0-1754486141061.png

 

Alex Coope - SN
ServiceNow Employee
ServiceNow Employee

@jain0811, you might have also accidentally copied the page level script as well for that kind of error to be thrown, or if the artifact wasn't made in the Global scope that might also be an issue,

 

Failing that, you can install the POC bundle from the store here,

Many thanks,
Kind regards
 

jain0811
Tera Explorer

@Alex Coope - SN , thanks for the quick response.

I’ve verified both points you mentioned:

  • The page-level script was not copied accidentally.

  • The artifact was created in the Global scope as expected.

However, I’m still encountering the same error when clicking Request Translation.

Could you please advise on the next steps?

Alex Coope - SN
ServiceNow Employee
ServiceNow Employee

Hi @jain0811,
Can you check if there's any "requested" RCA's (Restricted Caller Access) requests that need approving. As you're trying to translate the CSM portal, it might need access to certain widgets in that scope (based on the error),

Many thanks,
Kind regards

Timothy Onyskin
Tera Contributor

Also, go to Flow Designer and check the workflows that triggered.  If you are using the Dynamic Translation with a vendor, you can get more info from the step logs for "Translate text" (or it might be called something different.  I had that error at one time, and it was because my token wasn't getting auto-refreshed (so I wen t into the REST message/Authentication and triggered a 'Get Token').  Another time (not for the portal translation), it was because I was using the Google Translator and it has a limit of 5,000 characters per translation field (and I had too many).  If you have a large block of text on your portal and you are using Google as your translation provider, it could be that as well... but the Flow Designer check above will give you the actual error message.

jain0811
Tera Explorer

@Alex Coope - SN , 

I’ve checked Restricted Caller Access and provided the necessary approvals. I also installed the POC bundle from the store as suggested, but I’m still encountering the same error.

Any ideas on what else could be causing this?

jain0811
Tera Explorer

@Alex Coope - SN,

 

Any thoughts what else we can check here ?

Alex Coope - SN
ServiceNow Employee
ServiceNow Employee

Hi @jain0811,

 

At the moment, another avenue worth looking at (assuming the artifact has been made in Global, and all necessary RCA's have been approved), would be to comment out the loop that checks for "Portal Content" (it's the query to the [sn_cd_content_visibility] table), which appears to throw a similar error for instances where it's not been installed,

Many thanks,
Kind regards

jain0811
Tera Explorer

@Alex Coope - SN ,

It worked, Thank you so much for the help🙂

Version history
Last update:
3 weeks ago
Updated by:
Contributors