Importing ServiceNow Data into External Reporting Tool

[Robin Hearne]

I'm posting this question on behalf of a member of our data analytics team who is trying to pull ServiceNow data into his reporting toolset.  We've been working through several issues with HI support but we're being told that using SOAP to import data has a limitation of being able to retrieve only 2500 records at a time.  It seems like there should be a better way.  Anyway, his question is as follows:

I am using SAP BODS ETL tool to fetch data from ServiceNow connecting via webservice SOAP protocol. It has limitations of fetching only 2500 rows of data at a time.

My question to community is ..

What methods or ETL tools are you using to dump data from ServiceNow into other systems.

What type of connection used to fetch data from ServiceNow (like ODBC or webservice SOAP..etc)

Challenges/limitations faced while extracting data from ServiceNow

Regards,

Robin

[Pedro Lopez]

Hi Robin,

As mentioned from ServiceNow support, there is a limitation for importing data. An idea coming to my mind, you can create Rest Messages and run them once the record/data is created in ServiceNow to the External Reporting tool as part of an integration. This will be record by record but the external tool will have the data in the database and you can report based on that.

I hope this helps.

Regards,

Pedro Lopez

[sachin_namjoshi]

Hi Robin,

I have implemented similar requirement to extract data from Service now instance to oracle database.

I used data pump utility to implement this requirement.

You can find more details below

https://sourceforge.net/projects/servicenowpump/

Regards,

Sachin

[isaak_papagiann]

Hello Robin,

When I read the statement '2500 rows at at time', I assume it's a bunch data export and not a real-time export. Please remember when you fetch data using ODBC, ODBC also uses soap underneath and I have seen 100,000+ records retrieved successfully using ODBC.

Please remember that type of data being retrieved also matters and decides the number of records when retrieving data. Please try using ODBC if that answers your purpose.

[AnoopDK]

During my work to achieve an extraction of ServiceNow data trough ETL, I have learnt many things. However, I was 100% successful in pulling data from ServiceNow using ETL tool and stage it in Oracle DB. Later, I was able to apply data cleaning, conversions and mapping process to finally load into the reporting Oracle DB tables.

Here, I have used SOAP URL to connect to SN using WSDL file/link, aggregation and data-load SN calls, build ServiceNow query to pull data, Java code embedded in ETL process for various data validations and checks, used threshold levels to identify data losses during data load, email attachments to report every data load status and much more.

Achievements, have implemented more than 20 tables. This also includes, security layer design, reference fields mapping, building views upon tables for reporting and much more. ETL process can pull any number to data during load without any restrictions.ac

Limitations, above-all, there are still certain limitations that SN need to work on to make this ETL extractions seamless. Even few fields in tables have limitations in-terms of storing the maximum character data.

I welcome a discussion on why we need external ETL extraction process for building a dynamic real-time ServiceNow reporting. I have also worked on building solutions for SN reporting using BI tools. I can also provide excellent documentation for HLD and TLD.

Contact for consultation.