Fundamentals of Testing for Programmers

Testing is an important part of a software development. Testing is tightly coupled with development and many teams include not only developers but also testers (or even whole testing department). Because of this tight coupling, it is nice when testers and developers can share a common dictionary. With this goal, I'm posting here my notes on this topic.

In this post, I will try to highlight what is testing, how it's connected with quality assurance and quality management. And also will describe seven testing principles and will path you through a fundamental test process.

Why is testing necessary?

Testing of components, systems, and their associated documentation, can help reduce the risk of failures occurring during operations. When defects are detected subsequently fixed, this contributes to the quality of the components or systems.

Causes of Software Defects

A human being can make an error (mistake), which produces a defect (fault, bug) in the program code, or in a document. If a defect in code is executed, the system may fail to do what it should do (or do something it shouldn’t), causing a failure.

Defects in software, systems or documents may result in failures, but not all defects do so. A defect in the code, being executed, may cause failure, but not necessarily in all circumstances. For example, some defects require very specific inputs or preconditions to trigger a failure, which may occur rarely or never.

Errors (mistakes, defects) may occur for many reasons, such as:

• Time pressure
• Human fallibility
• Inexperienced or insufficiently skilled project participants
• Miscommunication between project participants
• The complexity of the code, design, architecture, problem etc
• New, unfamiliar technologies

In addition to failures caused due to defects in the code, failures can also be caused by environmental conditions.

Not all unexpected test results are failures. False positive may occur due to errors in tests, test data, test environment or for other reasons. The same things can lead to false negative. False negatives are tests that do not detect defects that they should have detected; false positives are reported as defects, but aren’t actually defects.

What is Testing?

A common misperception of testing is that it only consists of running tests. software testing is a process which includes many different activities; test execution is only one of these activities. The test process also includes activities such as:

• Test planning
• Analyzing, designing, and implementing tests
• Reporting test progress and results
• Evaluating the quality of a test object
• Reporting on the testing process and system under test
• Finalizing closure activities after a test phase has been completed
• Reviewing documents (including source code) and conducting static analysis

Another common misperception of testing is that it focuses entirely on verification of requirements, user stories, or other specifications. While testing does involve checking whether the system meets specified requirements, it also involves validation, which is checking whether the system will meet user and other stakeholder needs in its operational environment(s).

For any given project, the objectives of testing may include:

• To evaluate work products such as requirements, user stories, design, and code
• To verify whether all specified requirements have been fulfilled
• To validate whether the test object is complete and works as the users and other stakeholders expect
• To build confidence in the level of quality of the test object
• To prevent defects
• To prevent defects
• etc

Also good to note that testing and debugging are different. Executing tests can show failures that are caused by defects in the software. Debugging is the development activity that finds, analyzes, and fixes such defects.

Testing and Quality

With help of testing, it is possible to measure the quality of software in terms of defects found, for both functional and non-functional software requirements and characteristics.

Testing can give confidence in the quality of the software if it finds few or no defects. A properly designed test that passes reduces the risk of defects within the system.

People often use phrase quality assurance (or just QA) to refer to testing, but they are not the same (so they are related). Quality management is the larger concept that ties them together.  Quality management includes all activities that direct and control an organization with regard to quality and also includes such activities as quality assurance and quality control.
Quality assurance is focused on adherence to proper processes, in order to provide confidence in the appropriate level of quality. When processes are carried out properly, the work products are generally of higher quality, which contributes to defect prevention.

Quality control involves various activities, including test activities, that support the achievement of appropriate levels of quality. Since quality assurance is concerned with the proper execution of the entire process, quality assurance supports proper testing.

Seven Testing Principles

1. Testing shows the presence of defects, not their absence

Testing can show that defects are present, but cannot prove that there are no defects. Testing reduces the probability of undiscovered defects remaining in the software but it is not a proof of correctness.

2. Exhaustive testing is impossible

Testing everything is not feasible. Instead, risk analysis, test techniques, and priorities should be used to focus test efforts.

3. Early testing saves time and money

To find defects early, test activities should be started as early as possible in the software development lifecycle. Early testing is sometimes referred to as shift left.

4. Defects cluster together

A small number of modules usually contains most of the defects or is responsible for most of the operational failures. Predicted defect clusters and the actually observed defect clusters in test or operation are an important input into a risk analysis used to focus the test effort (as mentioned in principle 2).

5. Beware of the pesticide paradox

If the same tests are repeated over and over again, eventually these tests no longer find any new defects. To detect new defects, existing tests and test data may need changing, and new tests may need to be written. (Tests are no longer effective at finding defects, just as pesticides are no longer effective at killing insects after a while.)

6. Testing is context dependent

Testing is done differently in different contexts. For example, safety-critical industrial control software is tested differently from an e-commerce mobile app.

7. Absence-of-errors is a fallacy

Some organizations expect that testers can run all possible tests and find all possible defects, but principles 2 and 1, respectively, tell us that this is impossible. Further, it is a fallacy (i.e., a mistaken belief) to expect that just finding and fixing a large number of defects will ensure the success of a system. For example, fully testing all specified requirements and fixing all defects found could still produce a system that is difficult to use, that does not fulfill the users' needs and expectations.

Fundamental Test Process

A test process consists of the following main groups of activities:

• Test planning and control
• Test analysis and design
• Test implementation and execution
• Test completion activities

Test Planning and Control

Test planning is the act of defining the objectives of testing and specification of test activities in order to meet test objectives. On the other hand, control is the act of comparing actual progress against the test plan and reporting the status including deviations from the plan. Test planning takes into account the feedback from monitoring and control activities.

Test Analysis and Design

Test analysis determines “what to test” in terms of measurable coverage
criteria and includes next major activities:

• Analyzing the test basis appropriate to the test level being considered
• Evaluating the test basis and test items to identify defects of various types, such as ambiguities, omissions, inconsistencies, inaccuracies, contradictions, superfluous statements 
• Identify and prioritize test conditions
• Capturing bi-directional traceability between elements of the test basis and the associated test conditions

So, test analysis answers the question “what to test?” while test design
answers the question “how to test?" and includes next major activities:

• Designing and prioritizing test cases and sets of test cases
• Identifying necessary test data to support test conditions and test cases
• Designing the test environment and identifying any required infrastructure and tools
• Capturing bi-directional traceability between the test basis, test conditions, test cases, and test procedures

Test Implementation and Execution

So, test design answers the question “how to test?” while test implementation answers the question “do we now have everything in place to run the tests?”.

Test implementation includes the following major activities:

• Developing and prioritizing test procedures, and, potentially, creating automated test scripts
• Creating test suites from the test procedures and (if any) automated test scripts
• Arranging the test suites within a test execution schedule in a way that results in efficient test execution
• Building the test environment and verifying that everything needed has been set up correctly
• Preparing test data
• Verifying and updating bi-directional traceability between the test basis, test conditions, test cases, test procedures, and test suites

Test Completion Activities

Test completion activities collect data from completed test activities to consolidate experience and any other relevant information. Test completion activities occur at project milestones such as when a software system is released, a test project is completed, an Agile project iteration is finished etc.

Test completion includes the following major activities:

• Checking whether all defect reports are closed, entering change requests or product backlog items for any defects that remain unresolved at the end of test execution
• Creating a test summary report to be communicated to stakeholders
• Finalizing and archiving the test environment, the test data, the test infrastructure, and other testware for later reuse
• Handing over the testware to the maintenance organization
• Analyzing lessons learned from the completed test activities to determine changes needed for future iterations, releases, and projects
• Using the information gathered to improve test process maturity

Summary

In this post, I've tried to cover the basics of testing in software development. We've learned what is testing, how it's connected with quality assurance and quality management. Also, we've learned seven testing principles and went through a fundamental test process. I hope you liked it. Thank you for reading!