Automated Indicator- Showing Same Total for All Days

jlaps · ‎08-15-2017

Good morning community,

I am hoping someone can point me in the right direction with odd results I am getting with an automated indicator that should be counting all open incidents given the criteria in my indicator source. And it DOES work, however, every day it runs, it overwrites MOST previous days with the results from the last day. It makes it impossible to chart over time, and it is driving me nuts that I am probably missing something easy that is right in front of me.

Indicator Source-

Indicator-

Results-

What am I doing wrong? I want a running total of the open INC inventory so I can chart it, but this gives me a straight line rather than the steady upwards progression we are trying to halt. Where is the bit of logic I am missing?

Thanks!

Dennis R · ‎08-15-2017

Woot! Sounds like we made progress!

As for collecting historic data, what you'll need to do is change your filter to account for the date. To do that, instead of saying Date Created between 1/1/2017 and 12/31/2017, change it to Date Created At or after 1/1/2017 00:00:00 [and] Date Created Before Tomorrow.

When you do that, as the historic collection iterates over each date, it will plug in midnight of the collection date in as "Tomorrow".

That should get you the data you need in a historic collection.

[Edit:]

Just to expand on what I typed above because it might look a bit arcane, when you run a historic collection, what PA does is iterate over each date that the collection runs for; for example, the past three months. It will run a collection for May 15, then May 16, then May 17, and so on, up through August 14.

For each date, it will generate a collection period start and collection period end. For May 15, the collection period start will be 2017-05-15 00:00:00, and the collection period end will be 2017-05-15 23:59:59. For July 3, the collection period start will be 2017-07-03 00:00:00, and the collection period end will be 2017-07-03 23:59:59. And so on.

When the collection actually happens, all dates and times are calculated relative to the collection period, not the current date. If you have dates hard-coded in the condition, such as between 2017-01-01 00:00:00 and 2017-12-31 23:59:59, then the collection period will be ignored, and all of your numbers will be the same (the current value).

If, on the other hand, you use a relative date, such as "Today", "Tomorrow", or the like, then the condition will apply to the date/time relative to the collection period. That's why when you use "Date Created At or after 2017-01-01 00:00:00 [and] Date CreatedBefore Tomorrow" as your condition instead of the between condition you were using, what happens is that the collector says, "Query for anything created at or after 2017-01-01 00:00:00 relative to my current collection period", which is 2017-01-01 00:00:00 since that's a literal, static value, "and created at or before Tomorrow relative to my current collection period", which, for July 3 for example, would be before midnight on July 4.

The end result will be the number of records opened at or after 2017-01-01 00:00:00 and before 2017-07-04 00:00:00 (that is, through 2017-07-03 23:59:59). On May 17, the effective range would be anything opened at or after 2017-01-01 00:00:00 and before 2017-05-18 00:00:00 (that is, through 2017-05-17 23:59:59). And so on.

Hope this helps!

--Dennis R

View solution in original post

jlaps · ‎08-15-2017

Dennis,

Here are the two jobs, one the usual, the other a historic I added this to for grabbing past days. Not sure if you can see the settings on these you want to-

Thanks!

Jeff

Dennis R · ‎08-15-2017

Can you check the jobs themselves? In particular, what is Run set to under Job parameters, and what are your Collection parameters for each? What I'm thinking is that you might be running a job daily with a relative interval that's longer than a day, in which case each collection will effectively overwrite previous collections up to whatever your relative interval is.

jlaps · ‎08-15-2017

Dennis,

I think that is the issue- my normal jobs were running 90 days each night. I changed it to 1, and I did get a new score for yesterday. How then do I collect historic data without overwriting the old scores? I want 227 back days .

Thanks,

Jeff

Dennis R · ‎08-15-2017

Woot! Sounds like we made progress!

As for collecting historic data, what you'll need to do is change your filter to account for the date. To do that, instead of saying Date Created between 1/1/2017 and 12/31/2017, change it to Date Created At or after 1/1/2017 00:00:00 [and] Date Created Before Tomorrow.

When you do that, as the historic collection iterates over each date, it will plug in midnight of the collection date in as "Tomorrow".

That should get you the data you need in a historic collection.

[Edit:]

Just to expand on what I typed above because it might look a bit arcane, when you run a historic collection, what PA does is iterate over each date that the collection runs for; for example, the past three months. It will run a collection for May 15, then May 16, then May 17, and so on, up through August 14.

For each date, it will generate a collection period start and collection period end. For May 15, the collection period start will be 2017-05-15 00:00:00, and the collection period end will be 2017-05-15 23:59:59. For July 3, the collection period start will be 2017-07-03 00:00:00, and the collection period end will be 2017-07-03 23:59:59. And so on.

When the collection actually happens, all dates and times are calculated relative to the collection period, not the current date. If you have dates hard-coded in the condition, such as between 2017-01-01 00:00:00 and 2017-12-31 23:59:59, then the collection period will be ignored, and all of your numbers will be the same (the current value).

If, on the other hand, you use a relative date, such as "Today", "Tomorrow", or the like, then the condition will apply to the date/time relative to the collection period. That's why when you use "Date Created At or after 2017-01-01 00:00:00 [and] Date CreatedBefore Tomorrow" as your condition instead of the between condition you were using, what happens is that the collector says, "Query for anything created at or after 2017-01-01 00:00:00 relative to my current collection period", which is 2017-01-01 00:00:00 since that's a literal, static value, "and created at or before Tomorrow relative to my current collection period", which, for July 3 for example, would be before midnight on July 4.

The end result will be the number of records opened at or after 2017-01-01 00:00:00 and before 2017-07-04 00:00:00 (that is, through 2017-07-03 23:59:59). On May 17, the effective range would be anything opened at or after 2017-01-01 00:00:00 and before 2017-05-18 00:00:00 (that is, through 2017-05-17 23:59:59). And so on.

Hope this helps!

--Dennis R

jlaps · ‎08-16-2017

Thank you both, Arnoud and Dennis, for your assistance on this. I had a mental block on the historical aspect; basically, the system is only so good at pretending it is at previous days. It is not blind to data that is already there. I was stumped on why it worked for reporting and when testing on current values. Thanks to you both, I think I have my head wrapped around how this should work. Thanks again!

Jeff