Import 2 Million Records.

[NamrataJain]

Hi All,

I have a data in excel which has around 2 Million Records and I would like to import the data regularly.

I know I can have scheduled import, but I am worried that it will slow down my instance.

I do have thoughts on SOAP messages but I am not sure if the client can provide me the same or not.

Also I was wondering of having a JDBC connection but since the database is internal and Se

Can someone suggest me some other method?

Thanks in Advance.

Best Regards,

Namrata Jain

[Robert Chrystie]

Hi Namrata,

One thing that I would add to Chuck's recommendation is that ideally you won't need to import all 2 million rows regularly.   If you are able to use a JDBC connection, and the Database you are connecting to supports it, you can configure your import to only pull in the updated records.   Doing so would then only pull in the rows that have changed since the last time the import ran, which should help with performance, except for the initial load.

[Chuck Tomasi]

Good point Robert. That's in the best practices article I linked to. Hopefully Namrata is (or will) read through it - it contains a lot of great information that can prevent issues (many I have personally encountered).  

[kreddyT]

Hi,

I am using JDBC connection to pull 4 Million from a SQL server database(for the initial load). I was only able to pull only 1M at a time. Is there a way to pull everything in one load? Any property that i should set on MID server?

Thanks,

Keerthana.

[VaranAwesomenow]

Midserver generally timesout when pulling data in order of 500K or more with OOB timeout setting of 300 secs, so it needs to be changed to a value of in order of 1 hr for 1 million and 2 hrs for 2 million and so on.