7. Not in Scope: How to make sure your bug is valid

As previously mentioned, one of the most important details you should review in any request is the “In Scope/Not in Scope” section of the Tester Spec Document. This information will show you the parameters under which the test needs to take place.

Under the ‘Not in Scope’ section you will almost always see 2 things: ‘Usability Issues’ and ‘Unusual User Behaviour’, but what are they exactly?

Usability issues can be understood as suggestions that a tester might make towards improving the app. We understand the need to point out how a product might be made better, and on a certain level, we agree with our testers, which oftentimes make absolutely brilliant suggestions.

However, that is not what we are looking for. In a way, we are sorry to stifle your creativity, especially since your ideas more often than not would bring value to the app you are testing.

But 99% of the time your job on our platform will be to test the functions and options that are already built into the app not to give your opinion on it. Our clients require us to test a product in the exact way it is presented to us, they are not looking for new ideas. And as in any service, we must provide that which is requested of us.

We hope this does not sadden you, and we do encourage you to find places and platforms where you can explore your imagination and where your input will be valued.

We ourselves care deeply about the products we are testing and would want them to improve.

But we are content with testing and finding bugs which are also in itself a way to make a product better. And we here at Tester Work consider that as our mission.

Sometimes it might be hard to understand what exactly is a suggestion/usability issue anyway. Well, the simplest way to understand what we consider a usability issue is to think about the following terms:

• What is vs what isn’t a usability issue

While testing an app in isolation you should always concentrate on the functions that are already available. 

For example, you might find it inconvenient to have to open the ‘Settings’ menu in order to reach a ‘Log-out’ button so you might consider that a ‘Log-out’ button should be available outside the ‘Settings’ screen next to the ‘Settings’ button itself. Therefore you go ahead and submit a bug report that says there should be a ‘Log-out’ button in the Side Menu. In this case, instead of testing the app layout already present, you are suggesting that the app developer should change their app. In the end, you did not concentrate on what was but instead you concentrated on what you thought was missing from the app.

The exception to this way of thinking is when you are not testing an app in isolation, but instead, you are testing the same app on multiple platforms or types of devices. In that case, if a function/button/screen is available on one platform but not the other then that should be reported as a bug.

• Expectations

Another way some of our testers trick themselves into submitting a Usability Issue is by having preconceived expectations about how an app should work. This is tricky as most times expectations are correct and should be followed. 

For example, the expectation when you tap the “Terms and Conditions” button is to be redirected to a screen where you can view the Terms and Conditions of the app. However, expectations can also lead you to submit a false report. For example, an app might have a different layout than normal expectations and when tapping the “Terms and Conditions” button instead of a different screen loading a text field might expand on the same screen underneath the button containing the correct information. In this case, even though the normal expectation might have been for a different screen to open, the designers of the app opted for a different way to provide the same service.

A tester’s job, therefore, is to maintain an open attitude about the information he/she is receiving via the app and always think about if the action that was triggered is the correct one.

Abnormal user behavior is a bit of a general term that is hard to define but fairly easy to understand.

We as moderators usually consider as ‘abnormal’ any action that is made in bad faith, maliciously, destructively, or made in a forceful way to trigger a bug. Abnormal might mean anything from spamming a section or button in the hopes of triggering an error or freezing the app, overloading the device’s memory with unnecessary apps, or using unrecommended or illegal tools like Jailbreaking, Device Rooting, Location Spoofing, or any other malicious software.

Unusual user behavior can also refer to a bug that requires such complicated or numerous steps to reproduce that is borderline impossible for a regular user to ever reproduce under normal circumstances.

We know that most of our testers do not report such issues with ill intent, but every time you test an app you must try to behave in a way that is both logical and honest.