Exporting 100k Records - PAR?

[Ryan S]

With the switch to Platform Analytics and classic reports being deprecated, how should we plan to schedule an export of 100k records? We need it scheduled to send via email and PAR doesn't let us export to csv.

[Dr Atul G- LNG]

Hi @Ryan S 

Exporting or sending 100K records via email is not allowed. To do this, you would need to change the export compatibility and consider the current setup, but this could impact system performance.

[Ryan S]

Can you explain what you mean? It is possible with legacy reports to export over 100k records. I'm not sure what you mean by changing the export compatibility.

[tiagomacul]

The guidance provided by @Dr Atul G- LNG  is absolutely correct and aligns with core platform protection principles.

Attempting to bypass OOB limits to send 100k records via email is not just a technical challenge; it is a significant risk to system performance and operational stability.

From an Enterprise Architecture perspective, the following strategies are recommended to move beyond this limitation:

1. Validating the 'Performance First' Mindset As correctly noted, changing export compatibility to allow such high volumes via SMTP can impact the email node and overall instance responsiveness. In the context of Digital Transformation, an architect's role is to protect the platform's long-term health while enabling business outcomes.

2. Modern Alternatives for High-Volume Data Instead of forcing a legacy email-based export, the recommended 'TO-BE' state involves more resilient methods:

• Export Sets: This is the OOB standard for large datasets. It allows for scheduled exports to a MID Server, avoiding email size limitations and performance hits. * Table API or IDR: For organizations moving toward a mature data strategy, utilizing APIs or Instance Data Replication (IDR) to feed external BI tools (like PowerBI or Tableau) is the most scalable approach.

3. Strategic Governance Insight In the Now Create methodology, implementations should be Value-Oriented.

• Signal vs. Noise: 100k records in a CSV often indicate that users are performing data manipulation outside of ServiceNow.

• The Goal: Shift the culture from 'spreadsheet-led management' to 'in-platform insights' using Platform Analytics (PAR) dashboards. This ensures a 'Single Source of Truth' where data remains governed and secure.

Summary: While the deprecation of classic reports might seem like a hurdle, it is actually the perfect moment to retire 'Technical Debt'—like high-volume email exports—and adopt modern data consumption patterns.

For those looking to align their analytics and data strategies with official architectural pillars, the Now Create methodology offers a structured roadmap: ServiceNow Now Create: Practical Methodology"

[Ryan S]

As for Item 1, yes we're aware of potential performance issues and agree it should be avoided. (Even though I haven't found any hard data to support this claim.)

Export Sets may be the way we have to go, but it's cumbersome because getting the data onto the MID server is just step 1. Then we'd have to create some automated means to move that data at least to a network share. APIs and IDR won't work here as further explained next.

As for item 3 and the summary -- I don't disagree but how do you solve the problem of users needing to be able to further process data beyond the capabilities of ServiceNow? Perhaps we could implement it with extensive customization, but they can do exactly what they need in Excel already so why make it harder. Perhaps the data could be sent to a BI solution -- but that means standing up a BI solution. If the data is needed to provide inputs to external systems, then we could integrate to those systems. That's only if the system is modern enough to support such integration and if network/security and the system owner allows it. The data isn't being used for management or work within ServiceNow, so there is no advantage to leaving the data in ServiceNow and forcing the users into a system that wouldn't support their needs.

So I'm still left with wondering how ServiceNow intends us to extract large amounts of data using PAR. It appears PAR is intended for in-system analysis and I don't disagree with that purpose, but it's ignoring the full picture.