Seven Immutable Laws Of Testing
This post is intended to be QA humor. I am certain there are additional “testing laws” out there that need a mention. Seven seems to be the magical number that applies to successful habits, attributes, paths, processes, and pretty much anything and everything else that appears to be out of reach of the general population. Enjoy.
1. The bug that reproduces on your test environment every time, never reproduces on the developer environment.
I believe that almost every tester has faced the perplexing time when the bug readily reproduces on the test environment but is rejected by developer as “doesn’t repro on my computer.” I personally faced this issue at least twice when the developer convinced me that the bug was simply a test environment issue. He went on to place my test environment under his debugger and ran through the scenario successfully. You can’t argue when your favorite developer takes great pains to debug the issue and actually makes the scenario work as expected.
Naturally, the bug surfaced in production within minutes of the software release and we scrambled to release a hotfix.
To avoid a repeat scenario, my key learning from this experience is to reproduce the issue on a second test environment. This usually proves that the issue is likely real and is not environment related.
2. When that super critical hotfix needs to be tested ASAP, the test environment takes a dump and refuses to work.
I suspect that my test environment has a mind of its own. It knows when there is a hotfix coming down the pipe that needs to be tested ASAP. Soon as the hotfix is ready and deployed to the test environment, shit hits the fan. Even the basic functionality is broken rendering the test environment utterly useless.
There is no earthly explanation for this pesky phenomenon. Since time is of essence in such scenarios, I usually keep the test environments that I manage up-to-date and ready with the latest database snapshots and developer releases. If one of the test environment gives me grief, it is usually time optimal to simply use another one instead of fixing the broken one.
3. The automation that showed a high pass rate all throughout the sprint, is almost always hosed with massive failures in the final run before the release.
I believe that this testing law is related to the one mentioned earlier. The test environment knows when a hotfix needs to be tested. The test environment also knows when the release day is. After all, our release pipeline is published on the wiki for all the company to see.
There is no easy solution to this issue except to re-run the entire automation again. This is, of course, assuming that there is really no product bug and the issue is entirely test environment related. Usually, I investigate the nightly automation runs early morning and simply re-run them if there is a hint of a test environment issue. This way, I have the re-runs finish up by early afternoon for a pre-release investigation and inspection.
4. The developer is on break/lunch/PTO right when you need her input on a mission critical issue.
We all take breaks, go out to have lunch or an occasional paid time off (PTO). Lack of availability of developer expertise is generally not an issue. Unavailability’s importance increases when that hard to repro bug finally surfaces and the test environment needs to be investigated.
5. A bug always surfaces on the only test scenario that you forgot to execute for the new feature.
I usually draw up the test scenarios and get them reviewed for completeness before I start executing them. Developers are pretty good at pointing out test gaps based upon their knowledge of the code. They have, after all, coded up the functionality and know the intricate inner working details.
In spite of all due diligence, sometimes bugs slip through the tight test net. It is not surprising that the bugs went undetected due to some test scenario that was not executed.
6. When your boss uses the new, and fully tested, feature for the first time, the feature falls apart immediately and you can't explain how or why it is broken!
This had happened to me more times I care to remember and almost every time the root cause of the bug was the special set of conditions that were present on the new environment.
When I owned setup and configuration of a server product at Microsoft, the setup installer failed on a fairly recent laptop because one of the product pre requisites was missing. We were testing so diligently with test environments with all pre-requisites installed that that particular scenario of missing pre-requisite was never given a thought.
I believe I truly understood the idea of beta testing with real customers. It is near impossible to re-create and test all possible combinations of factors that go into setting up and configuring a product.
7. When you file one issue capturing all small bugs related to the new feature, it makes it harder to spread the developer work around. When you file individual issues capturing each bug, you are wasting your and developer time.
This law is not really about testing but about the test process. it is a fine balance between finding the bugs and reporting the bugs specially when the new UI is expected to have a large number of issues or improvements. Writing an issue up takes more time than finding one.
I have learned to confer with the developers before filing a bunch of bugs as to what would work best for them. Generally, filing the same class of bugs as one issue is convenient for developers.
If there are specific test scenarios where the bug surfaces, then filing individual issues (with the test scenario as the repro steps) generally works better.
In any case, the best course of action depends upon the team dynamics and developer preferences and how far behind the project is on the schedule.
The above “testing laws” are solely based on my personal experience. What’s your favorite “Immutable Test Law?”