Tuesday, August 7, 2012

Why not to create Test Documentation

Before you read further, I would like you to do a small exercise. For each product/release that you have tested, calculate an approximate number for,
  1. Total no of Test documents you have written till date.
  2. Total no of test cases you've written.
  3. If it takes 5 minutes to write one test case along with the scripts, then total time consumed?
  4. Last time the scripts were updated

Once you are over the excitement of the numbers that you see in front of yourself and for those who could imagine how large the number would be and didn’t bother to calculate, answer yourself, what usually happens when the product/feature is released?

Usually, important/happy cases are documented in the regression pack, which is usually automated in many cases. From then onwards, the feature test documentation is not used anymore. Then why do we spend so much time on it? Let me try to answer this.

  • The most obvious reason is that the management has asked us to do so because the piece of documentation is getting the company money or “billing the client”. We all like to earn money but is that the end goal?
  • We might be writing it because this is what testers have been doing for past so many years. The problem with this is that no one questions what they’re doing and gladly accept what they have been taught by their seniors.

What do we usually achieve when we try to follow the above process?

  1. Missed Deadlines because we were so busy writing test scripts and getting them reviewed from the customer that we forgot to test the product
  2. Heated discussion with the developers regarding why they did not mention it in the FSD in the first place?
  3. A Buggy product
  4. An unhappy end user and customer

If the above points doesn’t seem so convincing to you, then ask yourself, “Do I like to write test documentation?” If your answer is still Yes, then I hope god is with the product that you have tested. 

What could you do instead of wasting time in test documentation?
  • Take time in understanding the requirements. Create a mind map.
  • Have a question & answer round with the requirements analyst.
  • Have a debate with the Developers
  • Test to explore the product. Yes, I clearly meant to say, practice exploratory testing.

I don’t have a goal to change the world, but if you are passionate about testing and want to test the product instead of wasting your time in writing how to test the product, then do think about it.

Friday, April 27, 2012

Evidence of Testing

Recently the client, for whom I am testing, asked me whether our team records any evidence or not. I didn’t understand what they were trying to say, so I asked them what they meant by evidence. They told me that every test case should have a screenshot of the system under test when executing, so that they know we have actually tested the product. Perhaps they thought of us as 'checkers'  rather than 'testers'. They also might just wanted to see that if we were performing the tests correctly or not. 
Who would confirm that? Would it be the Dev team? This seems un-necessary because the test cases/scripts written are reviewed by the client. So any incorrect test case/script would have been rectified at that point itself.

We usually write pass/fail against each test case and I knew that it was more than enough ‘evidence’ to show that we did the execution. Why did they require evidence? Did they not believe us that we were actually testing the system and not just looting them?
On second thoughts, it might be possible from their perspective. Just imagine if your product has been sent thousands of miles away from you to be tested by persons whom you have never met in your life, you would definitely have a doubt. Even if you can’t imagine, you have to give the benefit of doubt to them.
So we can say that there was a trust deficit between the client and the customer, which is not at all good. Trust comes only from experience and not by evidence.

The point I want to make is not just about trust, it’s more about gathering evidence. Should testers be actually gathering evidence? Isn’t pass/fail enough?
I don’t think that evidence is necessary. Firstly, it’s boring, time consuming and tedious task. This is not the reason why testers became testers, to prove that they actually perform the tests. The amount of time a tester devotes to gather evidence, in that time you can perform more tests, whether scripted or not and actually benefit the product from it.

Let’s assume that you do gather evidence and attach screenshots along with the test cases. What benefit can we achieve from this exercise? The only answer that I can think of is none. There’s none benefit at all. Even if a bug does come up in UAT or a customer site, what will you do with the evidence?? The bug has to be fixed anyways. The blame will be on the testing team, but isn’t the testing team blamed every time when a bug is raised. Even if you had passed the test cases in SIT, then also the trust would vanish, the primary reason for which the evidence was taken.

The other disadvantage of gathering evidence is the maintainability. In every cycle of execution, you would have to maintain every version’s screenshots, which again is a tedious task.

Let’s say that capturing evidence is a necessary evil. What are the possible solutions to filter it down and not make it a tedious exercise?

1)      Get a screen recorder which can record the steps that you are executing.
2)      If you can’t afford one, then take evidence of only those test cases that are high priority. (Still a useless process)
3)      Ask the client for more time. They would definitely think about it again.
4)      It should only be done in the final test execution cycle before the release.