Instance management - best practise

[Guled Yusuf]

Hi, the team and I discussed general best practices today but did not agree on a specific topic.

What is the right way to handle emergency updates on production instances?

Example - The update set is moved to production and for some reason, the change applied needs to be turned off until a complete fix is ready in the next update set.

1 - Is it right to just deactivate the introduced component in production? Like an ACL rule that was introduced.

2 - Or should you not deactivate it manually and just push a new update set where this ACL is deactivated?

Will there be any downsides if we manually deactivate the introduced component directly in production? Example Business rule, new ACL, etc. This would be the fastest way, but we need to understand if it goes against best practices and why. 

Thanks!

[Allen Andreas · Accepted solution]

Hello,

Some would say that you would backout the update set, but I don't necessarily agree with that unless the update set in question only contains that one piece. I actually created a video on my YouTube channel about the hidden risks of backing out the set.

Anyways, you can easily just deactivate the record for now, if it's sincerely causing problems, then have the team work on it properly and promote the record back up. When you go to commit it again in Prod, you'll most likely see a preview warning telling you that the record in Prod has been adjusted and so you'll just want to accept remote update anyway and it'll bring up your change successfully.

[Allen Andreas · Accepted solution]

Hello,

Some would say that you would backout the update set, but I don't necessarily agree with that unless the update set in question only contains that one piece. I actually created a video on my YouTube channel about the hidden risks of backing out the set.

Anyways, you can easily just deactivate the record for now, if it's sincerely causing problems, then have the team work on it properly and promote the record back up. When you go to commit it again in Prod, you'll most likely see a preview warning telling you that the record in Prod has been adjusted and so you'll just want to accept remote update anyway and it'll bring up your change successfully.

[Allen Andreas · Accepted solution]

@Guled Yusuf 

To follow-on to the guidance shared above, by me, and below, by Mark...you'd want to revisit this issue with your teams to find out why it may have happened in the first place.

• Was proper development not done previously?
• Was it not tested?
• Were the lower environments not in sync with Prod so there was a difference in behavior?
• Something else?

You'd still want to revisit the process cause of the issue and try and correct that as the most important and long term outcome here.

As I mentioned above, if an issue is causing a critical issue that's impacting your business, then all kidding aside...you need to deactivate the issue. Period. If it's not that urgent and can wait until later in non-peak hours, then consider having your team work on a "hotfix" to quickly address it...even if it's passing up an update set that contains the deactivation of that record (business rule, client script, etc.). Then, later, they can resume working on it in Dev and again bring it up properly.

I think you get the idea with deciding to make that decision in Prod or not based on the urgency/criticality.

[Guled Yusuf]

Thank you for the great answer, Allen. This is exactly what we discussed. We fixed it by deactivating it and then pushing an update set that fixes the issue, deactivation was done since it was critical at the time. 

Our disagreement was more about why we should not deactivate components in production. Some people are of the nature that nothing should be changed in production, while I do not see this as an issue since the deactivation is on an isolated and newly introduced component and it will close out the critical issue right there so we get time to push a long-term fix. That is why we were interested in hearing about the pros and cons.

[Mark Roethof]

Hi there,

In general, you would not do such manual development activity in production. Even deactivating a ACL, Business Rule, etc. = development. At some companies this won't even be possible since you wouldn't have admin rights in production. At other companies you simply always work with admin in production 😅

While in general you would not do such manual in production, sometimes its also being practical. If its urgent, end user impact, integrations broken, or takes a lot of extra effort to deactivate it, does that weighs against manually deactivating which costs you just a few seconds?

Personally I would always try to push back such manual work, even if its just a few seconds. For example already because of the mindset. Don't just develop in production. If you allow this, its really easy to start having a mindset o we can just do this in production directly, avoid that becoming the standard!