Save yourself from audit nightmare if you use Flow Designer

[JC S_]

If your organization is heavily using Flow Designer or you are just starting to fully maximize it on important workflows then you should know that:

ServiceNow, by default, permanently deletes your Flow Execution details after 6 months

Now, let's be more specific because for step reports, ServiceNow deletes them permanently just after 1 week!

Flow reporting data tables

Table	Description	Default retention period
sys_flow_action_report	Flow engine context action reports	6 Months
sys_flow_context	Flow context	6 Months
sys_flow_flow_report	Flow designer executions	6 Months
sys_flow_step_report	Flow engine context step reports	1 Week

SOURCE: Flow execution details retention | ServiceNow Docs - Quebec

How did we know?

Well, we learned it the hard way when we needed to troubleshoot a recently closed requested item and was surprised to find out that the Flow Execution details for that RITM was already permanently deleted by ServiceNow. It was very hard and nearly impossible to troubleshoot especially if the issue is related on the timing of how a specific action took place or if you have a frequently changed Flows/Subflows and since ServiceNow does not keep Flow versions as well (unless you manually copy it for every updated version you do) then you're screwed.

The worst part here is that ServiceNow did not document this behavior until New York! So for 3 releases (Kingston, London and Madrid) ServiceNow did not bother to inform their customers about this retention policy. It is not mentioned even once on any Now Learning courses related to Flow Designer and is not part of any administration training available as of February 2021.

What should you do?

If you already know about this retention policy and adjusted them prior - congratulations! You've saved yourself from a lot of audit headaches (How did you know about it? Let us know in the comments below). If this is the first time, you're hearing about this then stop whatever you're doing and do the following right away (or talk to your instance administrator to review).

It's quite simple, you just need to open the sys_auto_flush table, filter with Tablename starts with sys_flow_ then decide any of the actions below on resulting records:

1. Set the "Active" flag to false
2. or Set the "Age in seconds" value to maybe 10 or 15 years (depending on your organization's retention policies)

As ServiceNow warns on the documentation page, flow designer reporting takes a large amount of data for storage, so do this at your own risk. But let me ask you, if it's a question of whether you get screwed during your audits versus having system logs consuming storage space on your ServiceNow instance, what would you choose?

Closing thoughts...

6 months retention policy for such an important piece of system log is such an ill-thought design for Flow Designer. How an important piece of information like retention policy for Flow Designer was only documented after 3 major family releases is such a big oversight that hopefully ServiceNow will improve upon moving forward in terms of communicating important things like this (right now, this information is deeply buried in ServiceNow Docs portal.)

ServiceNow should really take a hard look at how these logs are stored in the system because there are numerous approaches that they can implement to tackle the large consumption of storage issue where this is all rooted. We can convert the execution details to a static snapshot for example or do a heavily compressed archiving system where the logs are moved off after set period but can be requested to be checked out ad-hoc for audit purposes.

Until ServiceNow comes-up with a better way on handling this mess back-end, learn from our experience now and save yourself from an audit nightmare.

PS. Please do vote this submitted idea on the Idea Portal so we can nudge ServiceNow to take action and address this issue.

[Allen Andreas]

Hello,

I can understand your frustration, but just to mention this...for the legacy workflow editor, those contexts are also deleted at 6 months. So this is a pretty standard length they've done for years and years. If this is something you're being audited on, and knew about, I would imagine your process manager or others who think of these things, would ask about those retention periods. It may not your job, but an audit manager, compliance, etc. someone should have asked about it.

So yes, could ServiceNow blatantly point to retention periods, yes, but, for an experienced system administrator, architect, process manager, these are things that would need to be brought up and looked in to no matter what system this is about: ServiceNow, Salesforce, etc.

Sorry for the bad experience you've had though, but it's not fully ServiceNow's fault.

Thank you for bringing this up to a larger audience to look out for others, but, I just wanted to cover that this is not something ServiceNow hid.

[JC S_]

Hi @Allen A,

Let's get this out of the way, we're not taking away any responsibility from our organization's resources/processes involved here.

Our organization went straightaway to and adopted Flow Designer for driving our request workflows. We came from handling requests via the Incident Management application which retains audit history for every action that have happened in the incident record not just for 6 months which became our reference point in terms of audit history retention.

ServiceNow has a responsibility to properly document important aspects of its product as part of its commitment to its customers. While the gap here may be unintentional (who would want their customers to suffer right?), the oversight in product documentation is clear as day.

Damage has been done on our end and we just want other customers to avoid this same issue. "Works for you" has been ServiceNow's tagline for quite some time and unfortunately on this specific instance, it did not.

We've already connected with ServiceNow folks that are working on this domain and we're positive that they will be able to address this soon (via training, documentation, and product/platform enhancements). Win-win for everyone involved in the Now platform.

[Allen Andreas]

There are also audit logs for RITMs as well that show most actions involved. Using incident management for "requests", which I hope you mean incidents, but either way, would use the same auditing then since normally there is no flow. So if you're auditing records for approvals, state/stage changes, and the like, most if not all of that is available...you just have to know the relationship between it all.

What specifically do you need from the context that you can't get from these other methods? Just curious... 

I don't want future readers confused here thinking that if they use flow designer for RITM...after 6 months...everything audit wise is gone, because that's absolutely NOT true.

[JC S_]

We have complex workflows that generates multiple catalog tasks some parallel, some with dependency, which may have specific catalog variables that gets updated and in turn triggers different actions in the workflow like inline sripts, subflows, outbound emails, REST actions, etc. Also, some of our RITMs lifecycle are more than 6 months (imagine having the request recently closed and right away you lose the execution details).

If you're using Flow designer to do basic actions especially if it covers approvals and journal and other RITM/SCTASK field changes only, then you're covered by the same audit history and M2M relationships provided by/linked to the task table but if you wanted to know the context on how a specific behavior in the workflow executed beyond the basics, then that's where the impact of this retention issue comes in.

Technically you can backtrack the Flow used on the catalog item, but some complex flows have, for example, subflows inside and subflows can be updated outside the main flow as such having that snapshot of the exact steps that was executed at what given time with what results becomes crucial.

For the usage intended for Flow Designer, execution details are particularly important. You may have only 1 task in the RITM and few field changes here and there, but hundreds of actions might be happening inside the Flow.

So yes, the important audit artifacts really do get deleted (permanently) after 6 months if one uses Flow Designer on RITMs - unless folks reading this act today and change the defaults as we shared above.

[adrian08]

I just found that we have multiple records of sys_flow_context:

Does it total? So 3m seconds + 1m seconds from the last sys_updated_on, that's when it gets deleted? 

[Richard Hine]

Adrian,

You will notice that two of the matchfields are sys_updated_on and if you look in them yourself you will see there are conditions when they will run.

The values you are showing match OOTB and you should not need to worry.

Richard

[adrian08]

@Allen Andreas  where are you basing your statement about the 6months retention for legacy workflow context? We have legacy workflow context as old are 9 years (completed workflows)