Testing in Python with unittest — theory with the sports centre, black box, white box, TDD and the pensioner who paid too much

At my university’s sports centre, a swimming pool ticket costs 2 € up to age 17, 5 € from 18 to 64 and 3 € from 65 on. I wrote the fee function for the booking website, tried it with a 10-year-old and a 30-year-old, and everything was fine. A week later a complaint arrived: a man who had just turned 65 had paid 5 €. The rules said “65 or older” and I had written age > 65. A single character. If I had written three tests instead of two, I would have seen it before anyone else:

import unittest


def fee(age):
    """Sports centre swimming pool ticket, in euros."""
    if age < 18:
        return 2
    if age > 65:             # the rules say '65 or older'...
        return 3
    return 5


class TestFee(unittest.TestCase):
    def test_adult(self):
        self.assertEqual(fee(30), 5)

    def test_child(self):
        self.assertEqual(fee(10), 2)

    def test_senior_exactly_65(self):
        self.assertEqual(fee(65), 3)   # the boundary value nobody had tested


unittest.main(verbosity=2)

Output:

test_adult (__main__.TestFee.test_adult) ... ok
test_child (__main__.TestFee.test_child) ... ok
test_senior_exactly_65 (__main__.TestFee.test_senior_exactly_65) ... FAIL

======================================================================
FAIL: test_senior_exactly_65 (__main__.TestFee.test_senior_exactly_65)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "hook.py", line 21, in test_senior_exactly_65
    self.assertEqual(fee(65), 3)   # the boundary value nobody had tested
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^
AssertionError: 5 != 3

----------------------------------------------------------------------
Ran 3 tests in 0.001s

FAILED (failures=1)

Two tests pass (ok) and the third one fails: fee(65) returns 5 and should return 3. That “5 != 3” is exactly the man’s complaint, but said by the computer and before going live. That is what this topic is about: how to choose the cases worth testing (black box and white box), how to automate them with unittest and how to work by writing the tests before the code (TDD).

We come from Python iterators and generators; after this topic you will practise with three blocks of tests.

What exactly is a test and what is it for?

A test means running a program with chosen data and comparing what it does with what it should do according to the specification. If they match, the test passes; if not, you have found a bug. Three words used carefully in FP2:

  • Error (mistake): what the person did. In the opening story, writing > instead of >=.
  • Defect: what the mistake left in the code. The line if age > 65:.
  • Failure: what you see from outside when running it. The 65-year-old paying 5 €.

And the most important idea in the topic, one that shows up in quizzes: testing finds errors, it does not prove there are none. If ten tests pass, you only know those ten cases work. That is why you do not test “at random”: you choose the cases most likely to reveal a defect. The two techniques that do that are black box and white box.

A good test is one that finds an error. A suite where everything is ok the first time is not necessarily good: it may only be testing the easy cases.

unittest — setting up and running tests in Python

Before seeing how to choose the cases, let’s see how to write them, because we will use them in every example. unittest comes with Python (nothing to install) and works like this:

  1. Import the module: import unittest.
  2. Create a class that inherits from unittest.TestCase (you saw it in Python inheritance).
  3. Write methods whose name starts with test. Only those run; a method called check_something will never be seen.
  4. Inside, use the class’s assert… methods to say what you expect.
  5. Run everything with unittest.main(). With verbosity=2 it shows one line per test with its name.

The opening example has the whole structure. When you run it, each test ends in one of three ways, and you should tell them apart at a glance:

ResultWhat it meansHow it shows
okEvery assert in the test heldA dot . in normal output; ok with verbosity=2
FAILAn assert did not hold: the code did something elseF, and a block with AssertionError
ERRORAn exception nobody expected: the test never got to checkE, and the traceback of that exception

Look at all three at once, using the padel court function we will meet in the white box section:

import unittest
from court import court_price   # the function from the previous example


class TestResultKinds(unittest.TestCase):
    def test_fails_wrong_expected(self):
        self.assertEqual(court_price(2, False, False), 15)   # 2 * 8 is 16: the test is wrong

    def test_ok_one_hour(self):
        self.assertEqual(court_price(1, False, False), 8)

    def test_raises_with_text(self):
        self.assertEqual(court_price('2', False, False), 16)   # '2' * 8 is a string


unittest.main(verbosity=2)

Output:

test_fails_wrong_expected (__main__.TestResultKinds.test_fails_wrong_expected) ... FAIL
test_ok_one_hour (__main__.TestResultKinds.test_ok_one_hour) ... ok
test_raises_with_text (__main__.TestResultKinds.test_raises_with_text) ... ERROR

======================================================================
ERROR: test_raises_with_text (__main__.TestResultKinds.test_raises_with_text)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "fail_error.py", line 13, in test_raises_with_text
    self.assertEqual(court_price('2', False, False), 16)   # '2' * 8 is a string
                     ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "court.py", line 11, in court_price
    if price > 20:                 # node 6 (predicate)
       ^^^^^^^^^^
TypeError: '>' not supported between instances of 'str' and 'int'

======================================================================
FAIL: test_fails_wrong_expected (__main__.TestResultKinds.test_fails_wrong_expected)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "fail_error.py", line 7, in test_fails_wrong_expected
    self.assertEqual(court_price(2, False, False), 15)   # 2 * 8 is 16: the test is wrong
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 16 != 15

----------------------------------------------------------------------
Ran 3 tests in 0.001s

FAILED (failures=1, errors=1)

Read the report from top to bottom:

  • The first three lines are a summary: one test per line, in alphabetical order of name (fails, ok, raises), not in the order you wrote them.
  • Then come the details: first the ERRORs and then the FAILs. The ERROR shows a traceback ending in TypeError inside court.py: we passed the text '2', '2' * 8 gives '22222222' and comparing a string with 20 blows up.
  • The FAIL ends in AssertionError: 16 != 15: first what came out, then what you expected. Here it is the test that is wrong (2 × 8 = 16), not the function. A test can have bugs too.
  • At the end, Ran 3 tests in 0.001s and the verdict FAILED (failures=1, errors=1). If everything passes, it says OK.

The time (“in 0.001s”) changes on every run and on every computer; you will get another number. And unittest writes its report to the error output (stderr): it looks the same in the terminal, but if you redirect the output with >, the report will not go to the file.

The most important assert methods

MethodPasses if…Example
assertEqual(a, b)a == bassertEqual(fee(30), 5)
assertNotEqual(a, b)a != bassertNotEqual(t1, t2)
assertTrue(x) / assertFalse(x)x is true / falseassertTrue(name.isalpha())
assertIn(a, b)a in bassertIn('padel', courts)
assertIsNone(x)x is NoneassertIsNone(bookings.get('tuesday'))
assertAlmostEqual(a, b)a and b match when rounded to 7 decimal placesassertAlmostEqual(0.1 + 0.2, 0.3)
assertRaises(Error)the with block raises that exceptionwith self.assertRaises(ValueError): fee(-5)
import unittest


class TestAsserts(unittest.TestCase):
    def test_assertalmostequal(self):
        self.assertAlmostEqual(0.1 + 0.2, 0.3)          # compares to 7 decimal places

    def test_assertequal(self):
        self.assertEqual(3 * 5, 15)

    def test_assertin(self):
        self.assertIn('padel', ['tennis', 'padel', 'squash'])

    def test_assertisnone(self):
        bookings = {'monday': 'Laura'}
        self.assertIsNone(bookings.get('tuesday'))      # get with no default value: None

    def test_assertraises(self):
        with self.assertRaises(ZeroDivisionError):
            1 / 0

    def test_asserttrue(self):
        self.assertTrue('Laura'.isalpha())


unittest.main(verbosity=2)

Output:

test_assertalmostequal (__main__.TestAsserts.test_assertalmostequal) ... ok
test_assertequal (__main__.TestAsserts.test_assertequal) ... ok
test_assertin (__main__.TestAsserts.test_assertin) ... ok
test_assertisnone (__main__.TestAsserts.test_assertisnone) ... ok
test_assertraises (__main__.TestAsserts.test_assertraises) ... ok
test_asserttrue (__main__.TestAsserts.test_asserttrue) ... ok

----------------------------------------------------------------------
Ran 6 tests in 0.000s

OK

Two of them deserve an explanation. assertAlmostEqual exists because 0.1 + 0.2 in Python is 0.30000000000000004 (decimals are stored in binary and are not exact): with assertEqual that test would fail even though the sum is “right”. And assertRaises is used with with: the test passes if that exception is raised inside the block, and fails if none is.

assertEqual(result == 5, True) or assertTrue(result == 5). They work, but when they fail they only say “False is not true”. assertEqual(result, 5) tells you what came out: “4 != 5”. Always use the most specific assert available.

White box vs black box — the real difference

Both techniques answer the same question (“which cases do I test?”) by looking at different things:

Black box versus white boxOn the left, the function as an opaque box: only inputs and outputs. On the right, the visible code with its ifs.Black boxWhite boxfee(age)?agepriceYou look at the spec:equivalence classesand boundary valuesif age < 18: return 2if age < 65: return 5return 3You look at the code:flow graph, pathsand cyclomatic complexity
They complement each other: black box finds what is missing; white box, what is extra or never runs.
Black boxWhite box
What you look atThe specification: what goes in and what must come outThe code: its ifs, loops and paths
FP2 techniquesEquivalence classes and boundary valuesFlow graph and cyclomatic complexity
When you can do itBefore writing the code (even for someone else’s code!)When the code already exists
What it finds bestCases the programmer forgot (the 65 in the opening story)Code that never runs or badly written branches

They do not compete: they complement each other. Black box does not know the code has a strange if; white box does not know a case required by the specification is missing (if the programmer never wrote it, there is no branch to walk).

Black box — equivalence classes

An equivalence class is a group of inputs the program should treat the same way. If the function works with age 30, it is reasonable to assume it works with 31, 40 or 57: they are in the same class. So instead of testing 121 ages, you test one representative of each class.

Classes come from the specification, and there are two kinds:

  • Valid: correct inputs that produce a result. In the fee: child (0-17), adult (18-64) and senior (65-120).
  • Invalid: wrong inputs that must give a controlled error. Here: negative ages, over 120 (the website does not accept more) and anything that is not a whole number.
Equivalence classes and boundary values of feeAge line: below 0 invalid; 0-17 child 2 euros; 18-64 adult 5; 65-120 senior 3; above 120 invalid. Boundaries -1/0, 17/18, 64/65 and 120/121.fee(age): which ages behave the same?invalid< 0EC1 child2 €EC2 adult5 €EC3 senior3 €invalid> 120-1|017|1864|65120|121Coral 64|65 is the opening bug: with > instead of >= age 65 paid 5 €. Plus EC6: a string (TypeError).
One value inside each band and both sides of each boundary: that is where bugs hide.
import unittest


def fee(age):
    """Pool ticket: age 0-17 2 €, 18-64 5 €, 65-120 3 €.
    TypeError if the age is not an integer; ValueError if it is outside 0-120."""
    if type(age) is not int:         # type() and not isinstance(): True would also be an int
        raise TypeError('the age must be a whole number')
    if age < 0 or age > 120:
        raise ValueError('age out of range')
    if age < 18:
        return 2
    if age < 65:
        return 5
    return 3


class TestFee(unittest.TestCase):
    # One representative value per equivalence class (EC)
    def test_ec1_child(self):
        self.assertEqual(fee(10), 2)

    def test_ec2_adult(self):
        self.assertEqual(fee(30), 5)

    def test_ec3_senior(self):
        self.assertEqual(fee(80), 3)

    def test_ec4_negative(self):
        with self.assertRaises(ValueError):
            fee(-5)

    def test_ec5_over_120(self):
        with self.assertRaises(ValueError):
            fee(150)

    def test_ec6_not_integer(self):
        with self.assertRaises(TypeError):
            fee('thirty')

    # Boundary values: both sides of each boundary
    def test_lim_invalid(self):
        for age in (-1, 121):
            with self.subTest(age=age):
                with self.assertRaises(ValueError):
                    fee(age)

    def test_lim_valid(self):
        for age, price in ((0, 2), (17, 2), (18, 5), (64, 5), (65, 3), (120, 3)):
            with self.subTest(age=age):     # if one fails, the report says which
                self.assertEqual(fee(age), price)


unittest.main(verbosity=2)

Output:

test_ec1_child (__main__.TestFee.test_ec1_child) ... ok
test_ec2_adult (__main__.TestFee.test_ec2_adult) ... ok
test_ec3_senior (__main__.TestFee.test_ec3_senior) ... ok
test_ec4_negative (__main__.TestFee.test_ec4_negative) ... ok
test_ec5_over_120 (__main__.TestFee.test_ec5_over_120) ... ok
test_ec6_not_integer (__main__.TestFee.test_ec6_not_integer) ... ok
test_lim_invalid (__main__.TestFee.test_lim_invalid) ... ok
test_lim_valid (__main__.TestFee.test_lim_valid) ... ok

----------------------------------------------------------------------
Ran 8 tests in 0.000s

OK

Look at how the tests are organised:

  • The first six (test_ec1 to test_ec6) are one per class: 10 years (child), 30 (adult), 80 (senior), −5, 150 and the text 'thirty'. The number in the name is not decoration: since unittest sorts them alphabetically, this way they come out in the order of the class table.
  • The three invalid classes use with self.assertRaises(...): what is checked is that the function complains with the right exception (ValueError for a number out of range, TypeError for something that is not a number).
  • The function uses type(age) is not int and not isinstance(age, int) because of a Python detail: True is an int (it is worth 1), and with isinstance the age True would pass as 1 year old.

Testing only the “happy path” (the valid classes). A good black box suite usually has more tests of wrong inputs than of right ones, because that is where the programmer did not think.

Black box — boundary value testing

Bugs are not spread at random: they pile up at the boundaries between classes, exactly where there is a <, <=, > or >=. The opening story is the perfect example: with 10, 30 or 80 everything worked; the failure only appeared at exactly 65.

The boundary value rule: for each boundary, test both sides. The fee has four boundaries, so you get eight values: −1 and 0, 17 and 18, 64 and 65, 120 and 121. They are the last two tests in the program:

# Boundary values: both sides of each boundary
def test_lim_invalid(self):
    for age in (-1, 121):
        with self.subTest(age=age):
            with self.assertRaises(ValueError):
                fee(age)

def test_lim_valid(self):
    for age, price in ((0, 2), (17, 2), (18, 5), (64, 5), (65, 3), (120, 3)):
        with self.subTest(age=age):     # if one fails, the report says which
            self.assertEqual(fee(age), price)

self.subTest(age=age) puts several cases in one test: if one fails, the report says which ((age=65)) and the others keep being checked. Without subTest, the first failing assert would cut the loop and you would not know whether the following ones fail too.

BoundaryLower sideUpper sideWhat it reveals
Start of range−1 → ValueError0 → 2 €< 0 written as <= 0
Child / adult17 → 2 €18 → 5 €< 18 written as <= 18
Adult / senior64 → 5 €65 → 3 €The opening bug (> 65)
End of range120 → 3 €121 → ValueError> 120 written as >= 120

With decimals, “the other side” is the closest value that makes sense for the problem: with grades to two decimal places, the boundary at 5 is tested with 4.99 and 5.

White box — the flow graph

Now we look inside the code. The price of the sports centre’s padel court: 8 € per hour, half price for members, 2 € per hour if the floodlights are on and a cap of 20 € per booking.

import unittest


def court_price(hours, is_member, with_lights):
    """Padel court hire, in euros."""
    price = hours * 8             # node 1
    if is_member:                 # node 2 (predicate)
        price = price * 0.5       # node 3: members pay half
    if with_lights:               # node 4 (predicate)
        price = price + 2 * hours # node 5: 2 € per hour of lights
    if price > 20:                # node 6 (predicate)
        price = 20                # node 7: 20 € cap per booking
    return price                  # node 8


class TestCourtPrice(unittest.TestCase):
    # V(G) = 3 predicates + 1 = 4 basic paths
    def test_path1_nothing(self):
        self.assertEqual(court_price(1, False, False), 8)

    def test_path2_member(self):
        self.assertEqual(court_price(1, True, False), 4)

    def test_path3_lights(self):
        self.assertEqual(court_price(1, False, True), 10)

    def test_path4_cap(self):
        self.assertEqual(court_price(3, False, False), 20)


unittest.main(verbosity=2)

Output:

test_path1_nothing (__main__.TestCourtPrice.test_path1_nothing) ... ok
test_path2_member (__main__.TestCourtPrice.test_path2_member) ... ok
test_path3_lights (__main__.TestCourtPrice.test_path3_lights) ... ok
test_path4_cap (__main__.TestCourtPrice.test_path4_cap) ... ok

----------------------------------------------------------------------
Ran 4 tests in 0.000s

OK

To do white box testing you draw the code’s flow graph: a node for each instruction or block of consecutive instructions and for each condition, and an edge (arrow) for each way of getting from one to another. Nodes with a condition (an if, a while, a for) are called predicate nodes: two arrows come out of them, Yes and No. In the program’s code they are numbered in the comments.

Flow graph of court_priceEight nodes, three predicates (2, 4 and 6) and ten edges: four basic paths with their test data and expected price.YesNoYesNoYesNo12345678price = hours * 8is_member?with_lights?price > 20?returnBasic paths (V(G) = 4)1-2-4-6-81 h, no member, no lights81-2-3-4-6-81 h, member41-2-4-5-6-81 h, with lights101-2-4-6-7-83 h, nothing else2010 edges − 8 nodes + 2 = 43 predicates + 1 = 4
The coral nodes are the questions (if): each one adds a path.

The rules for building it, the same as in the FP2 notes:

  1. Consecutive instructions with no decision between them go in the same node.
  2. Each if is a predicate node with two exits. An if without else has a “No” exit that skips the block.
  3. A loop (while, for) is a predicate node with an arrow going back.
  4. With compound conditions (if a and b:), the strict approach is one predicate node per simple condition.

White box — cyclomatic complexity

Cyclomatic complexity, V(G), tells you how many basic (independent) paths the graph has, which is the minimum number of tests to go through every edge. It is computed in three ways, and all three must agree (a good way to check the graph is right):

FormulaIn court_price
Edges − nodes + 210 − 8 + 2 = 4
Predicate nodes + 13 + 1 = 4
Closed regions of the graph + 1 (the outside one)3 + 1 = 4

With V(G) = 4 you need four paths. A basic path is one that adds at least one edge that the previous ones had not used. For each one you choose data that forces it:

PathWhere it goesDataExpected priceSum
11-2-4-6-8 (all “No”)1 h, no member, no lights81 × 8 = 8
21-2-3-4-6-81 h, member48 × 0.5 = 4
31-2-4-5-6-81 h, with lights108 + 2 × 1 = 10
41-2-4-6-7-83 h, nothing else203 × 8 = 24 > 20, capped

They are the program’s four tests, and all four pass. Look at path 2: the price comes out as 4.0 (a decimal, because of * 0.5) and assertEqual(4.0, 4) passes because in Python 4.0 == 4 is True.

If the three formulas give different numbers, the graph has a mistake (almost always, a missing edge in an if without else). It is the quickest check you can do in an exam.

TDD — Test-Driven Development

In TDD (test-driven development) you write the test before the code, in a very short cycle repeated for each requirement:

  1. Red: you write a test for something the code does not do yet. You run it and it must fail (if it passes, either the test is useless or the requirement was already done).
  2. Green: you write the minimum code to make it pass. Nothing more.
  3. Refactor: you improve the code (names, duplication…) without changing what it does. The tests, still green, warn you if you break something.

The example: the sports centre gym’s 10-session card. We want use() to raise ValueError when the card is used up. First the test, with the class still lacking that check:

import unittest


class Card:
    def __init__(self, holder, sessions=10):
        self.holder = holder
        self.sessions = sessions

    def use(self):
        self.sessions -= 1            # does not check yet whether sessions are left


class TestCard(unittest.TestCase):
    def test_empty_raises_error(self):        # the NEW test, written before the code
        card = Card('Pablo', sessions=0)
        with self.assertRaises(ValueError):
            card.use()

    def test_use_takes_one(self):
        card = Card('Pablo')
        card.use()
        self.assertEqual(card.sessions, 9)


unittest.main(verbosity=2)

Output (red):

test_empty_raises_error (__main__.TestCard.test_empty_raises_error) ... FAIL
test_use_takes_one (__main__.TestCard.test_use_takes_one) ... ok

======================================================================
FAIL: test_empty_raises_error (__main__.TestCard.test_empty_raises_error)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tdd_red.py", line 16, in test_empty_raises_error
    with self.assertRaises(ValueError):
         ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
AssertionError: ValueError not raised

----------------------------------------------------------------------
Ran 2 tests in 0.001s

FAILED (failures=1)

AssertionError: ValueError not raised: the new test fails because the code does not complain yet, and the card ends up with −1 sessions. That red is a good sign: it proves the test detects the missing requirement. Now, the minimum code:

import unittest


class Card:
    def __init__(self, holder, sessions=10):
        self.holder = holder
        self.sessions = sessions

    def use(self):
        if self.sessions == 0:        # the minimum for the test to pass
            raise ValueError('card used up')
        self.sessions -= 1


class TestCard(unittest.TestCase):
    def test_empty_raises_error(self):        # the NEW test, written before the code
        card = Card('Pablo', sessions=0)
        with self.assertRaises(ValueError):
            card.use()

    def test_use_takes_one(self):
        card = Card('Pablo')
        card.use()
        self.assertEqual(card.sessions, 9)


unittest.main(verbosity=2)

Output (green):

test_empty_raises_error (__main__.TestCard.test_empty_raises_error) ... ok
test_use_takes_one (__main__.TestCard.test_use_takes_one) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.000s

OK

Two ok. Refactor? Not needed here: the code is already short and clear. In the practice you will do the whole cycle, iteration by iteration, with a Temperature class for the pool water.

setUp and tearDown — preparing the ground

When several tests need the same starting object, don’t create it in each one: unittest has special methods that run by themselves.

  • setUp(self): before each test. Ideal for creating new objects (self.card = Card('Laura')).
  • tearDown(self): after each test, pass or fail. To close files or clean up what the test left.
  • setUpClass(cls) and tearDownClass(cls): only once per test class, before the first test and after the last one. They need @classmethod. For things that are expensive to prepare, such as opening a connection.

This program prints when each one runs. So that the prints and the report come out interleaved in the real order, at the end we tell unittest to write to normal output (stream=sys.stdout) instead of the error output:

import sys
import unittest


class Card:
    """Session card for the sports centre gym."""

    def __init__(self, holder, sessions=10):
        self.holder = holder
        self.sessions = sessions

    def use(self):
        if self.sessions == 0:
            raise ValueError('card used up')
        self.sessions -= 1


class TestCard(unittest.TestCase):
    @classmethod
    def setUpClass(cls):              # ONCE, before all the tests in the class
        print('setUpClass: I open the connection to the register')

    def setUp(self):                  # before EACH test: a new card
        print('  setUp: new card for Laura')
        self.card = Card('Laura')

    def tearDown(self):               # after EACH test, even if it fails
        print('  tearDown: I throw away the used card')

    def test_use_takes_one(self):
        print('    test_use_takes_one')
        self.card.use()
        self.assertEqual(self.card.sessions, 9)

    def test_new_card_has_ten(self):
        print('    test_new_card_has_ten')
        self.assertEqual(self.card.sessions, 10)   # not affected by the other test's use()


# The report to stdout too, so it is interleaved with the prints
unittest.main(testRunner=unittest.TextTestRunner(stream=sys.stdout, verbosity=2))

Output:

setUpClass: I open the connection to the register
test_new_card_has_ten (__main__.TestCard.test_new_card_has_ten) ...   setUp: new card for Laura
    test_new_card_has_ten
  tearDown: I throw away the used card
ok
test_use_takes_one (__main__.TestCard.test_use_takes_one) ...   setUp: new card for Laura
    test_use_takes_one
  tearDown: I throw away the used card
ok

----------------------------------------------------------------------
Ran 2 tests in 0.000s

OK

Follow the order: setUpClass once; then, for each test, setUp → the test → tearDown. Since unittest writes the test name and waits until it finishes to print ok, the prints inside appear in the middle of that line. And most important: test_new_card_has_ten sees 10 sessions even though the other test used one, because each test gets its own new card from setUp.

How to test a class — order matters

Notice that in the previous output test_new_card_has_ten ran before test_use_takes_one, even though it comes later in the code. unittest runs tests in alphabetical order of name (n comes before u). That gives three tips for testing a class:

  1. Every test, independent. No test may depend on another having run before (or on what it left behind). If your tests only pass in one order, they are wrong. In the exercises you will see what happens with a list shared between tests.
  2. Start with what everything else uses: constructor and validation, then the methods that change the state, then those that query or show it (__str__). If you number the tests (test_i1_…, test_i2_…), the report follows that order and, when something fails, the first red one is usually the cause.
  3. One test, one idea. Three short tests (test_deposit, test_deposit_nothing, test_deposit_surpassing_limit) are better than one long test that tries everything: when it fails, the name already tells you what broke.

Naming a method check_deposit or try_balance. Without the test prefix, unittest does not even run it and the report happily says OK. If you see “Ran 0 tests”, check the names.

Common mistakes with tests

Comparing decimals with assertEqual. 3 * 1.10 is 3.3000000000000003; use assertAlmostEqual.

self.assertRaises(ValueError, fee(-1)). The brackets run fee(-1) before assertRaises can watch it, and the test ends in ERROR. Use with self.assertRaises(ValueError): fee(-1) or pass the function and the argument separately: assertRaises(ValueError, fee, -1).

Confusing FAIL with ERROR. A FAIL is an assert that does not hold; an ERROR, an exception nobody expected (often a bug in the test itself).

When you write a test’s expected value, work it out by hand or from the specification, never by copying what your function returns. If you copy the output, the test will say ok even if the function is wrong.

Visualise it

Python Tutor does not run unittest (it needs to run the whole program in one go), but you can use it to see the idea behind a test suite: a list of cases with their expected result and a loop that compares them. In Python Tutor, choose Python 3 and paste:

def fee(age):
    if age < 18:
        return 2
    if age < 65:
        return 5
    return 3


cases = [(10, 2), (17, 2), (18, 5), (64, 5), (65, 3)]
failures = 0
for age, expected in cases:
    got = fee(age)
    if got != expected:
        failures += 1
        print('FAILS', age, got, expected)
print('cases:', len(cases), 'failures:', failures)

Output:

cases: 5 failures: 0

What to watch:

  1. The cases list holds (input, expected) pairs: 17 and 18, 64 and 65 are the fee’s boundary values.
  2. On each lap, look at got and expected in the global frame before the if: that is what an assertEqual does.
  3. Change age < 65 to age <= 65 in the function and run it again: you will see failures go up to 1 on the 65 lap and FAILS 65 5 3 printed.
  4. unittest does the same, but it also separates each case into its own test, counts the results and prints the report.

Quick summary

ConceptIn one sentence
TestRun with chosen data and compare with what is expected
Black boxCases taken from the specification: equivalence classes and boundary values
Equivalence classInputs treated the same way: one representative is enough (valid and invalid)
Boundary valuesBoth sides of every boundary: −1/0, 17/18, 64/65…
White boxCases taken from the code: flow graph and basic paths
Cyclomatic complexityEdges − nodes + 2 = predicates + 1 = minimum number of paths
TDDRed (failing test) → green (minimum code) → refactor
unittestClass inheriting from TestCase, test… methods, assert… and unittest.main()
ok, FAIL, ERRORPasses; an assert does not hold; unexpected exception

Three ideas to keep:

  1. Testing finds errors, it does not prove there are none: choose the cases wisely (classes, boundaries, paths).
  2. Bugs live on boundaries: always test both sides of every < or >=.
  3. Tests run in alphabetical order and each one must work on its own: use setUp to always start clean.

In the Python testing practice you will apply all this in three blocks: black box with FP2 grades, white box with the sports centre shop discount and a full TDD of the Temperature class. After that, the syllabus continues with linked structures: Python LinkedList.

Next step: practiceTesting in Python with unittest — practice with FP2 grades, the shop discount and a TDD for the pool (and the display that said 82 degrees)

Did this article help you? Here is how you can support SergioLearns →

Sigue aprendiendo

Leave a Reply

Your email address will not be published. Required fields are marked *