Testing Guidelines

Introduction

There are a series of test that are required when developing software. These are called unit, component, integration and acceptance tests. Each test a different aspect of the development phase. They ensure the correct functionality of the system as well as the expected behavior in corner cases. This testing is separate and in addition to evaluation of the results for scientific validation. These test emphasize software quality and maintenance, and are of utmost importance to assure funders, beta-testers, and eventual community users that our products can be trusted. In addition, without tests, once a certain scale is reached the code becomes difficult to maintain as every bug requires long and difficult debugging sessions. Further, every time the code changes, bugs appears out of the blue when trying to do science. Tests tend to catch those bugs while still working on the code and before running it for science.

To get started we will focus mainly on functional testing, but also consider usability, security, and performance which we will add tests for in future releases. The end goal is full compliance testing for certification.

Tests should not be written after the development phase, but at least in synch. This ensures that the developer is not biased from their own code when writing test cases.

Testing is incremental and as we move from unit test to acceptance tests, there will be a sense of replication and retesting. Despite that all steps are necessary and important.

Testing is automated through a system called Travis. The CI team will integrate your testing code with Travis on GitHub. This system will eventually run all the tests from a pull request ( will not be done in a commit basis) and will report back whether the testing was completed correctly or not.

The document continues with a more detailed description of each type of test. In addition, links are provided in the end of the document with existing tests. Feel free to open issues with questions in the repos of the test when they are easily understandable!

Unit Tests

Unit tests test the behavior of a single method or function. They are supposed to provide different inputs and verify the output. The test developer has to keep in mind a few things when writing a unit test. First, it should test all the common case inputs, Second, if there are branches, the test should check all possible outcomes of the branch. Third, test for inputs that are invalid. The last one is very important because it avoids false positive answers from the software.

Let’s say, for example, that you develop a function that does addition, subtraction and division. Multiplication is not supported. The function takes as input two number and a word such as ‘add’, ‘subtract’, ‘div’. The function below is an implementation of that function:

def foo(number1, number2, operation):

    if operation == 'add':
        temp = number1 + number2
    elif operation == 'sub':
        temp  number1 - number2
    else:
        temp = number1 / number2

    return temp

As you can see there is no check for either multiplication or division.

A test should make sure that all arithmetic operations are correct, division with 0 produces the correct error message and a different word other than the three mentioned produce an error message. The test test looks like:

def test_foo():

    value = foo(1, 2, 'add')
    assert value == 3  # Success

    value = foo(1, 2, 'sub')
    assert value == -1  # Success

    value = foo(4, 2, 'div')
    assert value == 2  # Success

    with ZeroDivisionError:
        value = foo( 4, 0, 'div')  # Success

    with RuntimeError:
        value = foo(2, 4, 'mul')  # Failed

The last test failed because foo function does not check for other types of operations and produce the correct error message. We can fix this bug by running the test.

Furthermore, unit tests test only the lines of code that do not call or interact with other components, methods or functions. For example, we are trying to test how the code of a function reacts based on the output of a called function. Instead of actually calling, we mock the call and return well known values that would trigger and test different parts of the code.

For example:

def foo2(input1):
    ...
    value = foo(2, 3, 'add')
    ...

A test would do the following:

def test_foo2():

    mocked_foo = 5

    # test foo2 code

Integration Test

Integration tests test how different part of the system work together. That includes internal component interactions, component interaction with the outside world, function or method integration.

The purpose of these tests to verify that when multiple parts of the code work together they do what is expected. Based on the last example, instead of mocking the called function the test now allows the call. Based on the result of the test, we can conclude whether the integration was successful or not.

Integration tests can lead to component level testing. When testing the interaction of different components we execute again an integration test. Acceptance Test This is the final test for a system. It tests whether the system is working according to its requirements. This test will be performed before a release of the software system.

References

For more information about testing: Software Testing Fundamentals

Existing Unit tests

Unit tests exist in the Seals use case repo in the experimental/data-design branch.