ATF development best practices - ATF Test Suites scope & implementation testing

[jMarshal]

Hi Everyone - hopefully this is a quick one and I am already pretty confident in my understanding, but want to ensure I am not starting our ATF journey with a bad habit. We are just starting to use ATF and what we do here will inform what other developers do, going forward, when it comes to our ATF development and deployment strategy. I have 2 ideas that I would like to validate as "best practice" or "recommended practice":

1. As a recommended best practice, customer-built ATF Test Suites that are for the Incident Management module [table = incident] should be developed in the "Incident ATF Test" application scope. This is rather than developing the ATF Test Suite in "Global". The OOB Incident Management ATF Test Suites are in the "Incident ATF Suites" scope and the associated nav menu item relies on that scope (application namespace) for the list filter.
   
   ...so, you will want to develop the customer-built ATF Test Suites that are for the Incident Management module in the "Incident ATF Test" scope, so that they have the correct namespace, so that they will show in the list of available ATF tests, when using the  "Incident ATF Suites" nav menu item which comes OOB...correct?
   
   
2. If we are developing ATF Test Suites in sub-production (promoting to production through normal deployment), our post-implementation test should done by cloning our production instance to sub-production and running ATF there, and if it fails, the implementation of the ATF Test Suites which were developed is considered a failed implementation (should enact change management back-out plan or fix during implementation window) and this needs to be addressed (fixed) in sub-production (where it was developed)...correct?
   
   I'm less confident in this point, as I have noticed that most sources advise to develop/build ATF directly in production...but that causes problems when testing the Tests and Test Suites which you are building.
   
   We have experienced this situation, where ATF was running fine in our test instance, but then we brought it from development to production -- but when cloning production back to test, the ATF test suites fail because there are missing components that were not correctly included in the update sets.
   
   I now want to back-out...but not sure if I should back-out...or leave it and re-develop...or delete the records from production and then re-develop...

Any advice here would be appreciated. Again, this is our first rodeo in ATF - so we have no existing artefacts or precedent to consider. That is what I am establishing.

Thanks in advance! I give thumbs-up to all helpful answers and input, regardless if I can use it!

[Dan Covic2]

Important note: the ATF tests are not captured in the Studio IDE nor in the App Engine Studio. They are only captured in the default scope of the scoped app or any other scoped update set that you have created specifically for implementing ATF tests. So, if you're working with a CICD pipeline (application repository) you should not worry about manually moving/migrating the ATF tests that were implemented in the default scoped update set or any other specific scoped update set (with respect to creating ATF tests) to another sub-instance. The ATF tests in the scoped update set would be automatically moved to another instance like with other app files in the scoped app.