Plugin installation issue + best practice?

[wclim]

Hi all,

Just to share a unique issue that I encountered for one of the customers recently.

Given that I have 2 instances:

1. Test Instance: Plugin is installed
2. Prod Instance: Plugin is installed
Both instances are on same version, same patch Tokyo Patch 9 . Plugin installed are the same version (to be specific, the plugin is com.snc.cs_base_extension v1.0.0)

Issue:
In the test instance, the plugin added an account field to the cmn_location table, but in prod instance, the plugin did not add this field to the cmn_location table, which caused an error when trying to push an update set to production as customizations on this field is made. 

After some investigation with HI support, it turns out this behavior is due to the plugin being installed when the Test instance was in Sandiego, and afterwards upgraded to Tokyo. Where in the Prod instance, the plugin was installed when it is in Tokyo. There are some database changes in the plugin between Sandiego and Tokyo, even though the version number is the same. 

We thought of just importing the missing field from UAT to prod, but as we are not sure what other missing configuration are there in this plugin as well as all other plugins that might have been installed on different servicenow versions, we decided not to go for this approach as it seems risky (we found this one plugin installed 4000 application files in Test compared to around 2000 application files in Prod)

Luckily, the prod instance was not live yet, so we were able to reprovision the Production instance in Sandiego, do the plugin installations again, and then finally upgrade to Tokyo where we were able to move the update sets successfully.

Conclusion

To me this is quite worrying as now plugin installations seems like a higher risk activity compared to what I thought previously.

I'm thinking of following this best practice moving forward:

1. Ensure all SN instances are provisioned at the same time in the same version
2. Before installation of a plugin, ensure all SN instances are on the same version
3. Ensure that if a plugin is installed at the same time in all SN instances

If for some reason, the SN instances are not provisioned at the same time, then we should at least get the production instance and do the initial build directly in production, and then clone the production to dev/test instances when they get provisioned to keep everything in sync.

TLDR:

Installing a plugin on an instance in a lower version and then upgrading the instance to a newer version is different from installing the same plugin (same version) in an instance already in the newer version, even if both instance have the same ending version and same installed plugin versions.

But there is a disadvantage of this practice - since plugins cannot be uninstalled, one will need to be very careful of installing a plugin since it cannot be reversed.

Like to know what are your thoughts on this

[Jeff Currier]

Yes, this makes sense.  Generally it is best to apply plugins (and other changes) in the same order for each instance.  First in dev, then test, then production.  "Most" of the time order doesn't matter, but the best way to avoid surprises is to keep the order the same.  For this reason, cloning production back to test and dev between major releases helps ensure your tests are valid and removes those things that will never make it to production.  Just be sure to capture any in process configuration so you can re-apply it after the clone.