Please help to guide us as we foray into the world of automation (we've been doing unit testing for years but not behavior testing).
We're going to have the following rule:
All tests must be independent tests (ie be capable of being run regardless of what tests have run before). This will allow us to use "Dynamic Test Lists" (which will allow us to create a list of all tests that relate to XYZ business requirements).
The problem that I foresee is that each "test setup" will be quite complex and time consuming.
Let me explain... In order for a complex business process to be tested, there are perhaps another 4 or 5 complex business processes that have to happen first (in order that the correct data gets in the database). This means that before the test can even start, there will be 4 or 5 long time consuming tests that have to execute EVERY time a test need to run:
Setup:
- Create location hierarchy structure
- Import lots of users
- Import other related data
- Schedule events to happen
- Run a calculation process
The problem is that this setup could take maybe 2 or 3 minutes to execute to completion. And only then can we start the actual test that we want to conduct.
This cycle of "Setup" then "Test" is going to be repeated lots so I can see that the overall testing is going to be very slow (yes but faster than manual).
The way we do this manually is that we break the "each test must be independent". So the tester manually runs the setup and then runs many tests using the data that is setup. They have to use their intelligence to ensure that the data left over from the previous tests does not affect the current test (or is taken into account when checking to see if the results are correct (and it is possible).
I don't think we can break that rule (all must be independent) for automatic tests because we do need to be able to run tests in any order (using Dynamic Test Lists).
Suggested Solution
Instead of running several recorded tests to build up the required setup data, we get the setup script to login to a "Super Admin" area of the application. It can then restore the database to a known state and then run the test script.
So the cycle will be:
Test #1: Restore database, run test #1
Test #2: Restore database, run test #2
Question
Does anyone see any maintenance issues with this approach?
Are other pitfalls to watch out for?
Thanks in advance,
Ian.