Workflow data Fabric - connect for PDF and text files outside of ServiceNow

[ShreeP]

I need to provide solution where there are multiple data source outside the ServiceNow. PDF and text files are stored in Data Bricks/Data warehouse or it can be not standard locations. This is unstructured data. These documents has policy, EOS, EOL and SOX compliance information. Data need to be massaged . So we can validate CI against this data before insert into CMDB. Data size could be ~150 GB. What is the best solution for it. 

 Thanks in advance

[sachinvic]

@ShreeP 

Hi,

For this requirement, avoid loading the PDF/text content directly into CMDB. The better approach is to keep the documents in their existing repository and use Workflow Data Fabric (WDF) to connect ServiceNow to the external data.

Workflow Data Fabric is designed to provide access to structured, unstructured, and streaming data from sources inside and outside ServiceNow. Zero-copy connectivity can also avoid unnecessarily copying large datasets into the instance.

A practical architecture would be:

Create or update CMDB records only with the required structured attributes

For the PDF and text content specifically, first confirm where the files reside and whether the requirement is to search the documents, extract attributes from them, or continuously validate their content against CMDB. That will determine whether an External Content Connector, Document Intelligence, Zero Copy Connector, or another WDF capability is the right fit.

For - 150 GB, keeping the source data external and accessing it through the appropriate WDF pattern is generally preferable to replicating the entire dataset into ServiceNow. ServiceNow specifically positions zero-copy connectivity as a way to access external data without duplicating or storing it in the instance.

FYR: https://www.servicenow.com/platform/workflow-data-fabric/get-started.html

Helpful? Please mark the response as Accepted Solution ✔️ and close the thread — this helps future readers find the solution faster.

Regards,
Sachinvic
✨ ArchX Certified | ✨ 16x ServiceNow AI MVP

[ShreeP]

Dear Sachin, thanks for your suggestion 

Below are the challenges i faced during my solution consideration:

• Zero Copy Connectors - Limitation - it is that useful for structure data. As we are talking unstructured data for searching and making decisions
• Document Intelligence used only for individual new documents limit is 20 MB and 200 pages in flow.​
• XCC-Files up to 25 MB, and only about 1 MB of extracted text per file goes into the index. It indexes for search, not for extracting structured fields​
• Due to large volume it is not advisable to bring so much big data in to ServiceNow

[sachinvic]

Hi Shree,

Thanks for sharing these details. The limitations you mentioned are important for this use case.

Given the large volume of unstructured PDF/text data, I would avoid bringing the complete dataset into ServiceNow or using Document Intelligence for bulk processing.

You can instead keep the documents in the external repository and use ServiceNow for the required access, validation and workflow.

For structured external data, Zero Copy can still be considered where applicable. However, for unstructured documents, you should look at an external document-processing/extraction layer to identify and extract the required information, and then pass only the relevant structured data to ServiceNow.

This approach also avoids unnecessarily moving ~150 GB of documents into ServiceNow.

The key point is to separate document storage/extraction from ServiceNow validation and workflow processing.

FYR: https://www.servicenow.com/docs/r/zurich/intelligent-experiences/now-assist-in-document-intelli...

FYR: https://www.servicenow.com/docs/r/integrate-applications/prerequisites-data-products-wdf.html

Regards,
Sachinvic
✨ ArchX Certified | ✨ 16x ServiceNow AI MVP