Evidence of Testing

Recently the client, for whom I am testing, asked me whether our team records any evidence or not. I didn’t understand what they were trying to say, so I asked them what they meant by evidence. They told me that every test case should have a screenshot of the system under test when executing, so that they know we have actually tested the product. Perhaps they thought of us as 'checkers'  rather than 'testers'. They also might just wanted to see that if we were performing the tests correctly or not. 

Who would confirm that? Would it be the Dev team? This seems un-necessary because the test cases/scripts written are reviewed by the client. So any incorrect test case/script would have been rectified at that point itself.

We usually write pass/fail against each test case and I knew that it was more than enough ‘evidence’ to show that we did the execution. Why did they require evidence? Did they not believe us that we were actually testing the system and not just looting them?

On second thoughts, it might be possible from their perspective. Just imagine if your product has been sent thousands of miles away from you to be tested by persons whom you have never met in your life, you would definitely have a doubt. Even if you can’t imagine, you have to give the benefit of doubt to them.

So we can say that there was a trust deficit between the client and the customer, which is not at all good. Trust comes only from experience and not by evidence.

The point I want to make is not just about trust, it’s more about gathering evidence. Should testers be actually gathering evidence? Isn’t pass/fail enough?

I don’t think that evidence is necessary. Firstly, it’s boring, time consuming and tedious task. This is not the reason why testers became testers, to prove that they actually perform the tests. The amount of time a tester devotes to gather evidence, in that time you can perform more tests, whether scripted or not and actually benefit the product from it.

Let’s assume that you do gather evidence and attach screenshots along with the test cases. What benefit can we achieve from this exercise? The only answer that I can think of is none. There’s none benefit at all. Even if a bug does come up in UAT or a customer site, what will you do with the evidence?? The bug has to be fixed anyways. The blame will be on the testing team, but isn’t the testing team blamed every time when a bug is raised. Even if you had passed the test cases in SIT, then also the trust would vanish, the primary reason for which the evidence was taken.

The other disadvantage of gathering evidence is the maintainability. In every cycle of execution, you would have to maintain every version’s screenshots, which again is a tedious task.

Let’s say that capturing evidence is a necessary evil. What are the possible solutions to filter it down and not make it a tedious exercise?

1)      Get a screen recorder which can record the steps that you are executing.

2)      If you can’t afford one, then take evidence of only those test cases that are high priority. (Still a useless process)

3)      Ask the client for more time. They would definitely think about it again.

4)      It should only be done in the final test execution cycle before the release.