
- Subscribe to RSS Feed
- Mark as New
- Mark as Read
- Bookmark
- Subscribe
- Printer Friendly Page
- Report Inappropriate Content
Over the last few months I’ve spoken to several of my customers who have all asked me very similar questions regarding the use of Automated Test Framework, typically on the back of the discussions around platform upgrades:
- How do I use ATF to speed up upgrades?
- Will ATF replace the manual testing that we do? or
- How many tests should I create in ATF?
The premise appears simple: If I can write automated tests, this will negate the manual effort required to test as part of an upgrade. However, it’s not quite as clear cut as that, and this blog will go on to explain the influencing factors that you should consider as part of coming up with your ATF Strategy.
Why do we do testing?
We test our platforms ahead to time to mitigate against potential failure or adverse impact on our business. Our decision to test is to mitigate that risk.
In the case of ServiceNow upgrades, we want to ensure that our existing business approved configurations & customisations are preserved and operate in the same way after an upgrade or major enhancement.
The effort associated with testing
Testing, whether it’s manual or automated requires time (and therefore cost to your business) to establish and maintain in the future. Although ServiceNow provides starter ATF cases for most applications, there can still be significant effort required to modify these to suit your processes within ServiceNow.
ServiceNow OOB comes with a suite of ATF tests that are mapped against OOB processes and functions. Make copies of these tests and ensure they remain valid during your development lifecycle.
Subsequent enhancements and modifications you make to your ServiceNow platform will require a review [and often an update of] your ATF tests, it’s important to ensure your tests remain current so that they are valuable at the time of upgrade or major enhancement.
Tests can quickly become dated if unmaintained, so ensure that appropriate governance exists to mandate the test management as part of the development lifecycle. The level of detail you decide to go to for your automated tests will dictate the amount of ongoing effort and remediation that will be required to maintain those tests alongside all of your future development activity. Do not underestimate this effort.
What should we test?
I’ve seen instances where customers have built extensive ATF tests that cover every category & subcategory combination, and tests that have 20+ steps that test very specific scenarios. Are these wrong? Not necessarily.
I’d like to revisit why we test. We test to mitigate against the risk of something adverse happening. We want to ensure that the business can operate after a major change. Therefore, I like to recommend three broad categories to be considered as part of an automated test strategy:
- End to end process tests
Ensure that your processes function end-to-end using their positive process flow. For example, in Change management ensure that the Normal, Standard and Emergency change flows can run end-to-end and that approvals are generated in the correct sequence.
The idea here is that you are testing that the process can execute from start to finish using known-good parameters.
- Key business processes/steps
Items that are considered critical to the execution of the processes or running of the business should have ATF tests to cover them. For example, your Twilio integration as part of Notify may form part of your essential business process so you want to test that an Incident can be promoted to a Major Incident and that your Major Incident Team would continue to receive the notification post-upgrade.
These key features/functions must be of high value to the business, as for every enhancement you make to your platform will need to be checked to see if it impacts these key business processes tests.
- Security, privacy or compliance requirements
Any items that relate to data security or personal information should have an ATF test. For example, if there is a field that shows an employee’s date of birth, and this should only be visible to the Service Desk, I would write a test for that.
We want to ensure that these easy to test and high-value items are tested automatically.
- End-user impacting functions
You should test items within your service catalogue. Test the items that get used the most and the items that are most critical to your business. Don’t re-test the same logic across multiple service catalogue items as this is redundant.
Test to ensure that users can track their tickets and that they can search within the knowledgebase.
What should we consider not testing?
I write this section with a caveat, that it shouldn’t be taken prescriptively. There may be scenarios where you do want to test the following, but in general, I find it’s not needed.
- Data – Data is not likely to be impacted by upgrades, though I suppose it could be removed as part of a platform change that you would want to identify. However, will the process break without the data? Will it stop Incidents being logged? Perhaps you may write a test that ensures choices exist, rather than each choice exists.
- Process Exceptions/Errors – Automated testing requires investment, so you want to ensure the process runs, not that it doesn’t. Once you venture down the path of automated negative testing your test set will balloon out and become a nightmare to maintain.
- Mandatory Fields (generally) – New mandatory fields introduced during development can break existing tests. It is important to keep tests up to date when adding new mandatory fields, so I’ll only generally check for mandatory fields when it relates to a security or compliance requirement. These mandatory field checks can be cumbersome to maintain when you have conditionally mandatory fields.
How does automated testing fit into the broader testing landscape?
It’s important to remember that automated testing and the use of ATF is only one type of testing that should form part of your overall testing strategy. In addition to your ATF tests you should also consider the following types of tests to ensure you have a complete view of how a change impacts the system:
- User Acceptance Testing – Allow users to follow their procedures and work instructions to ensure that the system functions according to their requirements
- Exploratory testing – Allow users to go in and try to break the system. I find this the most interesting type of test, as a good test analyst can uncover unique situations that automated tests could never pick up
- Integration testing - To ensure all integrations and data loads operate correctly.
Collectively, these types of tests should give you the confidence to ensure that your platform changes won’t break once in production.
Conclusion
Unfortunately, there is no 1 size fits all when it comes to automated testing. A small organisation who uses OOB applications, versus large organisations who use ServiceNow as their ESM platform will have very different security, legal, compliance and governance requirements which dictate what the appropriate level of testing is for them.
While I haven’t been able to give a direct answer, I hope that this short article has given you some insight as to what may be required to get a good outcome and lasting results from your investment in ServiceNow’s ATF.
I’d be very interested in the thoughts of others with regards to ATF. Feel free to share your thoughts or test strategies in the comments below.
- 2,410 Views
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.