Data sources - what is "Use last run datetime" and how to use it?

[#1]

Data sources - what is "Use last run datetime" and how to use it?

The documentation on 'Use last run datetime' is a bit unclear.

example:

table: student
column: name
column: is_enrolled
column: total_points

What might make a suitable Last Run Table Field?

How does the system match rows from previous runs without knowing what the unique key is? on what criteria is it matching rows?

Thanks,
Afshin

[mdwallick]

In your example, there is no field suitable to use as a last run datetime. The purpose of this data source attribute is to allow you to use a date/time field from your source data as a way to only grab records that have changed since the last time you loaded the data source. It has nothing to do with unique identifiers or anything like that. Uniqueness is handled in transform maps using the coalesce attribute.

If your source data set is small, not using a last run datetime isn't a big deal. If you have a large data set (e.g. millions of records) then you definintely only want to get what's changed since the last load, not the entire data set.

Does that help? It's really pretty straightforward.