Recently the client, for whom I am testing, asked me whether
our team records any evidence or not. I didn’t understand what they were trying
to say, so I asked them what they meant by evidence. They told me that every
test case should have a screenshot of the system under test when executing, so
that they know we have actually tested the product. Perhaps they thought of us as 'checkers' rather than 'testers'. They also might just wanted
to see that if we were performing the tests correctly or not.
Who would confirm
that? Would it be the Dev team? This seems un-necessary because the test
cases/scripts written are reviewed by the client. So any incorrect test case/script
would have been rectified at that point itself.
We usually write pass/fail against each test case and
I knew that it was more than enough ‘evidence’ to show that we did the
execution. Why did they require evidence? Did they not believe us that we were actually testing the system and not just looting them?
On second thoughts, it might be possible from their
perspective. Just imagine if your product has been sent thousands of miles away
from you to be tested by persons whom you have never met in your life, you
would definitely have a doubt. Even if you can’t imagine, you have to give the
benefit of doubt to them.
So we can say that there was a trust deficit between the
client and the customer, which is not at all good. Trust comes only from
experience and not by evidence.
The point I want to make is not just about trust, it’s more
about gathering evidence. Should testers be actually gathering evidence? Isn’t
pass/fail enough?
I don’t think that evidence is necessary. Firstly, it’s
boring, time consuming and tedious task. This is not the reason why testers
became testers, to prove that they actually perform the tests. The amount of
time a tester devotes to gather evidence, in that time you can perform more
tests, whether scripted or not and actually benefit the product from it.
Let’s assume that you do gather evidence and attach
screenshots along with the test cases. What benefit can we achieve from this
exercise? The only answer that I can think of is none. There’s none benefit at all. Even if a bug does come
up in UAT or a customer site, what will you do with the evidence?? The bug has
to be fixed anyways. The blame will be on the testing team, but isn’t the
testing team blamed every time when a bug is raised. Even if you had passed the
test cases in SIT, then also the trust would vanish, the primary reason for
which the evidence was taken.
The other disadvantage of gathering evidence is the maintainability.
In every cycle of execution, you would have to maintain every version’s
screenshots, which again is a tedious task.
Let’s say that capturing evidence is a necessary evil. What
are the possible solutions to filter it down and not make it a tedious exercise?
1) Get
a screen recorder which can record the steps that you are executing.
2) If
you can’t afford one, then take evidence of only those test cases that are high
priority. (Still a useless process)
3) Ask
the client for more time. They would definitely think about it again.
4) It
should only be done in the final test execution cycle before the release.
