start After in 'Lookup objects by bucket stream' of amazon S3 is not filtering the data

[rohitservicenow]

I am trying to pull the files from Amazon S3 using the Amazon S3 spoke in Flow Designer. When I run the flow, it's listening all the keys from S3 bucket despite me adding the start after. 

Files in the bucket:

File_20230922202500.csv

File_20230922212500.csv

File_20230922222500.csv

File_20230922232500.csv

Start after: File_20230922222500.csv

When I run start after, I expect only the files after 'File_20230922222500.csv' to be brought back to Servicenow, but its listing all the files the bucket. 

Does anyone know how to fix it?

Thank you in advance,

Rohit

[msd93]

Hi @rohitservicenow 

The issue you're experiencing might be due to a misconfiguration or misunderstanding of the "Start After" parameter in the Amazon S3 Spoke in Flow Designer.

Here are some steps to troubleshoot and resolve the issue:

1. Verify the "Start After" parameter: The "Start After" parameter is used to list objects in the bucket that come after the specified key name in lexicographical order. Ensure that the key name you're providing is correct.

2. Check the order of the files: The files in your S3 bucket are listed in lexicographical order, not by the time they were uploaded. Ensure that the file you're using as the "Start After" parameter is not lexicographically last.

3. Update the IntegrationHub ETL: If you're using an IntegrationHub ETL to pull data from the S3 bucket, you might need to update the ETL to filter out the files you don't want.

4. Custom Scripting: If none of the above solutions work, you might need to write a custom script to filter out the files you don't want after the data is pulled into ServiceNow.

Please note that, the "Start After" parameter in the Amazon S3 Spoke is based on the lexicographical order of the keys, not the time they were uploaded. If the keys of your files don't follow a lexicographical order that matches their upload time, you might need to adjust your approach.

Hope this helps you.

[rohitservicenow]

Thank you @msd93

I have now arranged the files in Ascending order by Key in amazon S3 bucket and the files are still not getting filtered. I remember it working when I last tested on my PID a couple months ago, but not sure why it's not working now.