I am trying to understand how a process of remote file import and transform works. The scenario I am trying to understand is the following:
- Import a remote file (it contains new users )
- run a transform map to insert the new records into the user table
- execute this process on a scheduled basis
So, I created a remote file import scheduled job that imports a CSV file. Then created a transform map that takes the data from the import table and inserts user records. The job runs correctly at the scheduled time and imports the records into the import table. The transform (which uses the import table as the data source) works correctly as well, but I am running the transform manually. My question is, how do I run the transform map after the remote file import completes? Can I have the remote import trigger the transform? Can I schedule the transform? What I want to do is have the import and transform work a "unit".
Additionally, the import table gets larger every time the import runs since new records are added but the older records are not removed. How does the transform know to only take records from the latest import? And, what is the best practice regarding the older records? The table will keep getting larger and larger if it is not purged from time to time.
Ric
