madhadron

Excerpt from Easier, faster testing

How do we actually choose the conditions that we test?

For a function like _and, it’s easy. There are only four possible inputs:

We’ll test them all. But what about a function head that returns the first element of a list? We can’t test every possible list, and, intuitively, it would be incredibly wasteful even to try. So how do we choose which lists are important?

First, we use formal methods again. We annotate the types by creating a generic type variable with TypeVar.

from typing import List, TypeVar
T = TypeVar('T')

def head(xs: List[T]) -> T:
    if xs == []:
        raise ValueError('Cannot take head of empty list.')
    else:
        return xs[0]

The type of elements in the list is generic, so it doesn’t matter what type we use. We could even let T be NoneType, which has only one value (None), and use lists [], [None], [None, None], … .

That turns out not to be what we want because we would not be able to distinguish if we are accidentally taking the second or last element of the list instead of the first. For other functions, like finding the length of a collection, testing with NoneType is very useful.

For this case, though, we want a bigger type that we can distinguish values of, like int. But which values of List[int] are important?

This function only touches the first element of a list (if it exists). Passing a list of fifty one elements to this function is going to be basically the same as passing a list of fifty two elements or a hundred and two elements. On the other hand, the empty list, list with one element, and list with two elements are probably all things we want to test.

This is a pattern we will see again and again. If we consider the space of possible inputs to a function, most functions have small boundaries in that space where their behavior changes dramatically, connected by broad stretches where things are pretty much the same. This gives us an heuristic:

Principle 2: Carefully test the boundary. Sample the bulk.

With that in mind, let’s write a test plan for head.

import unittest
from head import head

class TestHead(unittest.TestCase):
    # Boundary:
    #   [] -> raises ValueError
    #   [522] -> 522
    #   [5, 2] -> 5
    # Bulk:
    #   [128, 53, 921, 1022, 41] -> 128
    #   [1, 5, 3, 12, 8, 1, 3, 2, 1, 5, 3, 19] -> 1
    def test_empty_list(self):
        with self.assertRaises(ValueError):
            # head behaves quite differently on [],
            # so we can't lump it in with the other
            # cases without being ridiculously
            # complicated.
            head([])

    def test_head(self):
        cases = [
            ([522], 522),
            ([5, 2], 5),
            ([128, 53, 921, 1022, 41], 128),
            ([1, 5, 3, 12, 8, 1, 3, 2, 1, 5, 3, 19], 1),
        ]
        for input, expected in cases:
            with self.subTest(input=input):
                found = head(input)
                self.assertEqual(expected, found)

This is the simplest case. The full course expands the boundary and bulk heuristic to multiple arguments, complex, recursive types and external services, along with introducing other techniques and heuristics.