Assertions, tests, and counterexamples.

A test checks a belief about a function, so write the belief down first. The belief is one sentence that says what the function returns, and for which inputs. Then write the tests. A test is a call with a chosen input, plus the value that the belief says the call should return, called the expected value. Work out each expected value from the belief, by hand, before you run the code. The test then compares two things made separately: what the code returns and what the belief says.

For each game, the app stores a score: the team’s points minus the other team’s points. So -1 means the team lost by one point, 0 means a draw, and 2 means the team won by two points. The belief: lowest returns the smallest score in a list that has at least one score. The tests use a small function, check(name, got, want), where got is the value lowest returned and want is the expected value. It prints pass when got equals want, and FAIL with both values when they differ. Its full code is in the experiment of the next section. check returns nothing, so it is declared void. count is the number of scores in the array.

Write down your belief and each expected value before you run the code.

Passing tests show that the belief holds for their inputs. They say nothing about other inputs. To learn more, look for a counterexample: one input for which the belief is false. One counterexample is enough to show that a belief is wrong, however many tests pass.

Look first at inputs the code was not designed around. Every list in the tests has a game the team lost or drew. A strong team can win every game.

Here lowest for {2, 1} returns 0, a draw the team never played, so {2, 1} is a counterexample. low starts at 0, and when every score is above 0, no score is below low. Keep the counterexample as a test. After the repair it passes, and it stays as a regression test, the kind of check that Minimal reproductions ends with.

Passing tests show that the belief holds for their inputs; one counterexample shows that it is false.

Counterexamples are most often found at boundaries: inputs at the edge of what the code handles. Try these first: an empty input; a single element; equal values; the answer in the first place and in the last; zero; negative numbers, and no negative numbers at all. A loop that starts or stops one place too early, or a starting value such as 0, is wrong at a boundary and right everywhere else.

For each boundary, work out the expected value from the belief before you run it. The belief may say nothing about an input. This belief says nothing about an empty list. That is a finding too: decide what should happen, and add the decision to the belief.

The repair starts low at the first score, scores[0]. The table in this section runs the repaired lowest on one input for each boundary. C has no empty array, so the empty list is a count of 0.

Try the edges first: empty, one, equal, first and last, zero, and negative.

A test that cannot fail still prints pass. It tells you nothing. Three kinds are common.

The first kind has no comparison. It calls the function and prints the result. Nothing in it can fail.

The second kind gets its expected value from the code under test. It calls the same function again, or it repeats the same formula. So it compares the code with itself.

The third kind has an expected value copied from the output of one run. It checks that the output does not change. If that output was wrong, the test keeps the bug.

The rule from the first section prevents all three: the expected value comes from the belief, worked out without the code. To check a new test, break the code on purpose. For example, make the function return 0 at once, and run the test again. If the test still passes, it does not test that code.

The test below gets want from lowest itself. The experiment runs it on the counterexample from the earlier section.

A test must be able to fail: its expected value comes from the belief, not from the code.

A test calls the code from outside, with inputs you chose. An assertion sits inside the code. It states that a condition is true at that point, every time the program reaches it. When the condition is false, the code itself is wrong, and the program should stop with a message.

Use assertions for programmer mistakes. A common one is a broken precondition: something that must be true when a function is called, such as “the list has at least one score”. The code that calls the function is responsible for it. Do not use assertions for problems that come from outside the program, such as a missing file, a score typed as text, or an empty list sent by another program. Those can happen when every line of the code is right. Handle them as errors and tell the user.

In C, assert(condition) comes from assert.h. It is a macro: a name the preprocessor replaces with code before compiling. When the condition is false, assert prints the file, the line, the function, and the condition to standard error, and stops the program with abort. When NDEBUG is defined before assert.h is included, every assert is replaced with ((void)0), an expression that does nothing, and the condition is not evaluated. Many release builds define NDEBUG, for example by passing -DNDEBUG to the compiler. For bad input, return an error code and let the caller decide.

Assert what only a programmer mistake can break, and build your tests without NDEBUG. Return an error code for everything else.

Predict the output, then look for a counterexample before you reveal the answer. A team’s score sheet shows the top scorer of a game, by position in the team list. The belief is the comment at the top of the program.