Executive summary
As part of any re-platforming effort, whether and how to migrate transactional data requires careful evaluation. The migration of transactional data from an original instance to a new instance should be avoided for the following reasons:
- High dependency and referential integrity risk: transactional records, for example Incident, reference foundation data that will have new sys IDs in the target instance, requiring comprehensive remapping.
- The audit trail is hard to preserve; journal fields depend on correct references, and detailed history is rotated, which makes a full audit difficult and often manual.
- Attachments inflate effort and storage: the sys_attachment and sys_attachment_doc tables are large, and migrating them increases the storage footprint of the new instance significantly.
However, there are always exceptions which may result in the need to migrate transactional data records. The migration of these exceptions, as well as the downstream impact on business processes, needs to be analysed in more detail as part of any re-platforming initiative.
Introduction
A core objective of any ServiceNow re-platforming effort is to leave behind years of accumulated technical debt (custom fields, bespoke workflows, non-standard configurations) and align to out-of-the-box (OOTB) processes on a clean new instance. Migrating transactional data carries a significant risk of undermining this objective by inadvertently reintroducing legacy artefacts, broken references, and inherited complexity into what was intended to be a greenfield environment.
This document is intended to support a data migration analysis discussion that is typically conducted during initial phases of a re-platforming initiative. It documents the general technical considerations involved, provides context for the recommended default position, and outlines a practical framework for handling exceptions.
Transactional data
Transactional data refers to records that capture individual events or activities within ServiceNow workflows. In other words, transactional data is defined as 'data recorded from the transactions within the platform'. Some examples of ServiceNow tables that contain transactional data include:
- Incidents (incident)
- Problems (problem)
- Changes (change_request)
- Tasks (task and its extended tables)
- Request Items (sc_req_item)
These tables typically have references to 'Foundation' or 'Master' data records such as Users (sys_user), Groups (sys_user_group), Locations (cmn_location) or Configuration Items (cmdb_ci). This foundation data differs from transactional data in that it is relatively more stable and does not regularly go through changes in state or status.
For the purposes of this discussion, the Incident table will be used as an example of transactional records. However, the principles discussed can be applied to other transactional record tables.
Referential Complexity
From an architectural perspective, the Incident table does not exist in isolation and contains a large number of references to other tables. This referenced data is required in order to have a complete Incident record with complete attributes. This means when considering migrating an Incident record, at least some data from all other referenced tables will need to be migrated as well, which greatly increases the effort required.
Figure 1: Out of the box references from the Incident table.
The image above illustrates the OOTB references from the Incident table to its referenced tables, all of which would need to be considered when migrating incident records. Furthermore, with legacy instances there are often other custom tables, not shown above, which are referenced by the incident table and therefore need to be considered for data migration also.
This challenge is compounded when foundation data is sourced from an external master data system and imported into the new instance, as it will naturally receive new globally unique identifiers (sys_ids). The result is broken referential integrity: a migrated incident record that references a legacy sys_id will no longer correctly resolve to the intended user, group, or CI in the new environment.
Example: An incident logged by user 'Jane Smith' in the legacy instance carries her legacy sys_id 'ABC123'. In the new instance, Jane Smith is imported from an HR or identity source and receives a new sys_id 'XYZ789'. Without remapping, the migrated incident record no longer resolves to Jane Smith.
Custom fields, custom values, and custom ACLs
In long-running ServiceNow instances (10+ years), transactional tables such as incident and the task derivatives accumulate custom fields, bespoke choice values, and ad-hoc ACLs that entangle process behaviour with historical design decisions. Lifting these artefacts into a new OOTB instance contradicts the goals of re-platforming, which are simplification, service-aware design, maintainability and upgrade safety. The default stance is therefore: do not migrate historical transactional data that depends on legacy customisations.
Regarding legacy ACLs specifically, migrating them can cause security ambiguity if they are not handled correctly. Legacy ACLs, covering create, read, write, delete and relation permissions, often no longer match the roles or data domains in the target instance. Moving them across can lock users out or overexpose data.
Where custom fields and values must be retained for business continuity reasons, the recommended approach is to carry them across as free text in a journal field or description field, so that the legacy schema is not recreated in the new instance.
While the challenges around custom fields, custom values, and custom ACLs are not insurmountable, migration requires many decision points, both business and technical, which adds further effort to the re-platforming initiative.
Journal fields and audit records
Journal fields
Within ServiceNow, journal fields take the form of the 'Additional Comments' and 'Work Notes' fields on task tables, such as the Incident table. There can be challenges with migrating journal fields because they operate differently to standard fields. Journal entries are append-only and reference the user who created them, so it is important to ensure all other referenced data, such as the user and group tables, is migrated prior to the journal fields. Migration sequencing is therefore an important consideration.
Audit records and table rotation
Audit records record when an update was made to a record, who made the update, and when the update occurred. This information is stored across multiple audit tables: sys_audit, sys_history_set, and sys_history_line. As explained in the following document, the difference between these tables comes down to persistence.
Auditing and history sets | How they work together
|
Audit table |
Behaviour |
|
sys_audit |
Records persist forever. |
|
sys_history_set |
Records are generated on use and are removed by the table cleaner 30 days after their most recent use. |
|
sys_history_line |
Records are held on four tables that are managed using Table Rotation. |
ServiceNow Table Rotation is a performance feature that splits large tables (such as syslog) into smaller, manageable physical tables (shards) and cycles them based on a set schedule, for example every 7 days, automatically deleting old data to maintain performance and control database size.
With regards to data migration, due to the nature of table rotation, getting a complete audit history for a record such as incident can be challenging to automate. Typically the detail of the audit, held in the history set line table, is overwritten due to table rotation and cannot be extracted. Often manual intervention is the only way forward, which can involve a large amount of effort.
Attachments
Frequently transactional records contain attachments which provide valuable context and information about that record. Unless attachments are migrated with the transactional record, there may not be enough information for that record to be triaged and resolved.
Migrating attachments involves two tables, sys_attachment and sys_attachment_doc, both of which are required to form a complete attachment record. These two tables combined are extremely large and contribute the highest portion of the ServiceNow storage footprint. Consequently, the migration of this data can take a long time and dramatically increase the storage footprint of the new instance.
Attachment migration: key considerations
- sys_attachment and sys_attachment_doc should always be migrated together, as a partial migration results in broken attachment references.
- Storage growth in the target instance should be modelled before committing to attachment migration.
- Consideration should be given to whether historical attachments on aged records genuinely need to be accessible in the new instance, or whether retaining the original instance as a read-only archive is a more proportionate approach.
Number maintenance
Number Maintenance is the mechanism in ServiceNow for managing the auto-incrementation and prefix of numbers on tables. If transactional data is to be imported from the original instance to the new instance, this would mean a decision around number maintenance would need to be made to prevent a collision of numbers. There are two common approaches:
- Offset the starting number. For example, if incidents have numbers ranging from INC0000001 to INC0010000 in the original instance, numbering in the new instance would need to start at a value greater than INC0010000 to avoid duplicates.
- Apply a different prefix. A prefix such as INCD could be used for migrated records, which has the added benefit of visually distinguishing legacy data from records created natively in the new instance.
Both approaches carry operational implications for reporting, integrations and user familiarity, and should be agreed with stakeholders before the import begins.
Exceptions
With every rule, there are always exceptions. It may be the case that there are reasons outside of a project team's control which require some transactional data to be migrated across to the new instance. Regulatory obligations, contractual commitments and in-flight records are the most common examples. By following some key recommendations and principles, the impact of this data migration can be kept to a minimum:
Exception principles
- The number of transactional records to migrate should be kept to an absolute minimum, and limited to those for which a clear, documented business or regulatory justification exists.
- Custom fields, values or ACLs should only be migrated where that documented justification exists. Where custom fields and values do need to be migrated, they should come across as free text in a journal field or description field, to avoid the proliferation of legacy custom fields and values.
- A consistent data-mapping approach needs to take place for all tables, with all referenced foundation data mapped from legacy sys IDs to new sys IDs before the transactional records are imported.
- Migrated data could store a reference or link to the equivalent record in the original instance for continuity.
Each exception should be documented with its rationale, scope and risks, and signed off by the appropriate business and technical stakeholders.
These principles and recommendations should always be discussed in more depth, taking into account each organisation’s unique context, to determine whether they are fit for purpose for the organisation's requirements.
Decision framework
The framework below provides a structured approach for evaluating whether a given class of transactional records should be migrated. It is intended as a starting point for the Transactional Data Migration Analysis activity rather than a definitive checklist.
|
Consideration |
Guidance |
|
Is there a documented regulatory or contractual obligation? |
Where one exists, it should be documented formally. This is the primary basis for an exception. |
|
Are the records still operationally active? |
In-flight records, such as open incidents and active changes, are stronger candidates for migration than closed or aged records. |
|
Can referential integrity be maintained? |
Confirm that all referenced foundation data will be present in the target instance, and that a sys ID mapping strategy has been defined. |
|
What is the attachment volume? |
Model the storage impact before committing. Consider archiving aged attachments instead of migrating them. |
|
Does the data depend on legacy custom fields? |
Assess whether free text capture is sufficient, or whether the cost of recreating the legacy schema is justified. |
|
What is the audit trail requirement? |
Determine whether a complete audit history is required, for example for regulatory review, and assess the feasibility of providing it given the limitations of table rotation. |
|
Is the original instance being decommissioned immediately? |
Where it is not, consider retaining the original instance as a read-only archive, which reduces the need to migrate historical records at all. |
Conclusion
Re-platforming to a clean, out-of-the-box ServiceNow instance is ultimately a modernisation and risk-reduction exercise. Migrating historical transactional data entangled with legacy customisations works against that aim by reintroducing technical debt, breaking referential integrity, and inflating storage and effort. Accordingly, the default stance remains: do not migrate transactional records. Where genuine business or regulatory exceptions exist, they should be narrowly scoped, deliberately planned, and executed with strong guardrails.
As a closing principle, the least-risk migration is the one that migrates the least data. Every transactional record brought into scope is a decision that should be justified, documented and approved.
