2. Severity: Some bugs are more valuable than others

Let’s talk about severity. When you submit an issue one of the main fields you will need to fill in is “Severity”. The severity of an issue is an often misunderstood grading system that many of our testers seem to struggle with. We hope we can clarify most if not all of those misunderstandings by showing you the way the system is implemented and why.

Let’s start by addressing the need for such a system in the first place.

Firstly, severity is defined as the extent to which a particular defect could create an impact on the application or system. This simply means that some bugs create a lot more problems than others and need to be given a higher priority when solving them. In the fast pace world of Software Development, time constraints and resource availability dictate that certain issues get a certain amount of attention. That’s why a grading scale needs to be implemented in order to maximize efficiency when solving bugs in the code.

The difference in severity can be easily understood when comparing a minor display issue like a spill or an overlap to something critical like a crash, but oftentimes that line can be blurry when comparing similar bugs with almost identical causes or effects.

Therefore, we hope this guide will help you understand the way we moderate bugs and set the correct Severity for yourself thus avoiding the correction penalty.

• Critical bugs:

In this category, we find bugs that have a major impact on the app, in a way that completely stops the user from accessing the app altogether or at least an important feature of the app.

A critical issue is anything that interrupts the user experience and makes them unable to proceed in the desired direction.

For example: ‘The app crashes when the user tries to log in”

In this example, the user is completely blocked from accessing any feature of the app. This is a definite show stopper and oftentimes blocks the entire test.

But one also has to consider the repro rate and possible workarounds. If this issue occurs every single time the user tries to log in and there is no way to bypass the crash, then the bug is indeed a critical issue.

However, if the issue only occurs once, and after a second or third attempt the user can successfully log in, then the issue is no longer critical. The path to the app is hindered but not blocked.

Another example of a critical issue is this: “When the user tries to perform any purchase, he receives an error message”

In this example, even though the entire app works perfectly, and the user can access all sections successfully, he is blocked from performing the purchase.

Therefore a good rule of thumb to remember is this: when the path of a user is completely blocked from accessing an important feature of the app, the issue should be marked as critical. Any workaround or low repro does not invalidate the issue, but it reduces its severity.

Following this line of thought, not all crashes are automatically critical issues. Any crash is definitely an unpleasant user experience and is to be avoided, but a random crash that can not be reproduced all the time, and that does not cause any important data loss is NOT critical by default. The same goes for crashes that can be easily avoided or are very specific in nature and only occur under extremely rare circumstances. 

For example, if a crash occurs only in a certain screen and to be reproduced the user needs to do a very specific series of steps that are not in any way required to perform the main function of the app, then that crash can become a high or even medium severity bug.

• High bugs:

Here we find bugs that have a deep impact on the app, but which do not stop the user completely from using a function of the app.

The line between critical and high bugs, therefore, needs to be understood as a function of how much access a tester has to the app and its sections. Any issue that limits or severely disrupts the user’s access can be labeled high.

For example: “An error occurs when accessing the Invite friends / Contacts page”. Here the user is hindered in using the full range of the social function of the app. But the issue is limited to an action in a subsection. It is an unpleasant experience but it does not stop the user from using the app for its intended purposes.

An important mention for all testers should be to firstly try and identify what is the purpose of the app. If the app is a social platform then the main functions of the app should be communication and chat systems. However, the app can be a game that just happens to also have social features. Therefore in a social chat app, when the user can’t access a friend, then that is a critical issue, but in the game app not being able to access a friend is a major but not critical one, as the main function of the game is not the social aspect but the game features.

Therefore testers should always pay close attention to the Spec Document that describes the main function and section/components of the app.

As a general rule for high severity bugs, this category includes all issues that impact the user in a way that hinders, limits, or greatly interrupts his actions and progress in the app, but which do not block his access completely.

• Medium bugs:

Medium bugs are issues that behave in a way that is different from the expectations of using the app. This may seem like a broad interpretation, but that is because most issues found during a test are of medium severity. This list includes any issue that affects the app in a minor functional way (as opposed to aesthetic), but the progress as a whole is not impacted.

This can mean any inconvenience or extra steps the user is required to face, but which can be overcome without great effort.

Example: “The ‘Next’ button has functionality only when tapped twice”

Here we can see that the user is forced to tap on the button twice in order to progress through to the next screen. Therefore the action is attainable but requires the user to perform an extra step that limits the fluidity of the user experience.

So in conclusion, a medium bug is any product defect that has a minor but not negligible impact on the intended functionality of the app.

• Low bugs:

These are any cosmetic defects including spelling mistakes, improper alignments, or font issues. In most cases, a low bug is anything that does not have any functional repercussions in the sense that something is merely unpleasantly displayed but functions properly.

This section usually also includes mistranslations, but the severity of a translation bug can vary based on the requirements of the test. That is why it is mandatory for testers to properly read the Tester Spec document before each test.

This means that the issue does not cause the user to perform any extra step to progress through the app, and the functionality is not impaired in any way.