Retain Import Set Data for a longer time - Bad idea?

[veena_kvkk88]

Hi Community leaders!!

I have a best practice question for you. We have some scheduled imports that run in various intervals and a while back, we changed the import set cleanup to clean records that are from 60 days ago (as opposed to the default 7 days - which did not result in any obvious issues yet)

For some of those imports, there were instances where data was not imported into the target table. We handle most of the possible scenarios but sometimes we cant figure out the reason why the import didn't happen. But if we have the staging table's record, that could give help us identify the root cause (Eg: Look at a field value that came in and realize that's the reason since we scripted to ignore the row if the string='Do not import')

But to do that, we need the staging table's data. So we are planning to extend the cleanup interval up to 6 months. Since this data is only required at the time of transformation and is not really queried anywhere, I assume this wouldn't give us any adverse affects.

But its still a very large amount of data - since, with the current 60day cleanup, we have about 3 million records. Your thoughts on this?

[sergiu_panaite]

60 days is a long period and staging tables can grow significantly during this period. Is the 3 million you mention for really 60day or default 7 day? It seems a low value to me for a 60 day period.

Have a look here as well:

Tips to improve the run time of imports

[veena_kvkk88]

Hi Sergiu!

Thanks for your response! The 3 million is the current number with the 60 day period. But sometimes we implement things a certain way if the requirement trumps best practice approach right? So if we need the staging data to validate the imports, is it still a total no-no on the extension or could we do it while keeping an eye out for the performance issues?