Before you begin
Why this matters
A function can pass every check you wrote and still be wrong for an input you did not think of. Write down what the function should do before you check it. Then look for the input that proves it wrong, and you find the bug before someone else does.
What this helps you reason about
- Write a check whose expected value comes from what the function should do, not from what it does.
- Choose the inputs where code is most often wrong: empty, one, equal, first and last, zero, and negative.
- Tell a check that could fail from one that always passes.
A team app shows each team’s worst game of a season. The function behind it has three tests, and all three pass. This lesson asks what those passing tests prove, and how to find the input that proves the function wrong.
Already know this? The short version
- Write down what the function should return, and for which inputs, before you choose inputs; every expected value comes from that sentence, worked out by hand, never from running the code.
- Passing checks prove that sentence only for their inputs; one input that gives a wrong result disproves it, so try inputs the code was not designed around.
- Try the edges first: empty, one element, equal values, first and last place, zero, negative, and no negatives at all.
- A check with no comparison, or whose expected value comes from the code it checks, cannot fail.
- Java’s assert does nothing unless the JVM starts with -ea; a check that must always run is an if that throws, and bad input is an exception the caller is expected to catch.
Words in this lesson (7)
- test
- Code that calls a function with a chosen input, compares the result with an expected value worked out in advance, and reports pass or fail.
- counterexample
- One input for which a belief about a function is false. A single counterexample is enough to show that the belief is wrong.
- regression test
- A check, kept with the code, that fails while a bug is present and passes once it is fixed, so the bug cannot come back unnoticed.
- boundary
- An input at the edge of what a function handles, such as an empty list, a single element, equal values, the first or last element, zero, or a negative number.
- assertion
- A check written into the program that a condition is true at that point. assert condition : message; throws an AssertionError when the condition is false, but only when the JVM was started with -ea.
- precondition
- Something that must be true when a function is called, such as “the list is not empty”. The code that calls the function is responsible for it.
- exception
- An object that is thrown to stop the normal flow when an operation fails, such as the NumberFormatException from Integer.parseInt. A catch block can handle it.
Examples are focused snippets; imports, enclosing functions, or classes may be omitted. C examples use C11; Haskell examples use Haskell 2010. Expand the notes for language-specific depth.
01 / The belief
What do you believe the function does?
A test checks a belief about a function, so write the belief down first. The belief is one sentence that says what the function returns, and for which inputs. Then write the tests. A test is a call with a chosen input, plus the value that the belief says the call should return, called the expected value. Work out each expected value from the belief, by hand, before you run the code. The test then compares two things made separately: what the code returns and what the belief says.
For each game, the app stores a score: the team’s points minus the other team’s points. So -1 means the team lost by one point, 0 means a draw, and 2 means the team won by two points. The belief: lowest returns the smallest score in a list that has at least one score. The tests use a small function, check(name, got, want), where got is the value lowest returned and want is the expected value. It prints pass when got equals want, and FAIL with both values when they differ. Its full code is in the experiment of the next section. for (int score : scores) visits each score in turn.
// lowest should return the smallest score of a list
// that has at least one score.
static int lowest(int[] scores) {
int low = 0;
for (int score : scores) {
if (score < low) {
low = score;
}
}
return low;
}
// Each line prints pass.
check("mixed", lowest(new int[] {1, -1, 2}), -1);
check("a draw", lowest(new int[] {0, 3}), 0);
check("one game", lowest(new int[] {-2}), -2);Write down your belief and each expected value before you run the code.
Coming from JUnit: assertEquals(1, lowest(new int[] {2, 1})) is the same test as check here, a call plus an expected value; in JUnit the expected value comes first.
Precisely: in a real project
JUnit 5 is the usual Java test framework. @Test marks a method as a test, and Maven (mvn test) or Gradle (gradle test) finds the test classes and runs every such method. assertEquals(expected, actual) throws when the two differ, which fails that test only. The expected value comes first.
This lesson uses a plain check function instead, because the stepper runs one file with no test tool.
import static org.junit.jupiter.api.Assertions.assertEquals;
import org.junit.jupiter.api.Test;
class LowestTest {
@Test
void allWins() {
assertEquals(1, Main.lowest(new int[] {2, 1}));
}
}02 / Counterexamples
Which input would prove the belief wrong?
Passing tests show that the belief holds for their inputs. They say nothing about other inputs. To learn more, look for a counterexample: one input for which the belief is false. One counterexample is enough to show that a belief is wrong, however many tests pass.
Look first at inputs the code was not designed around. Every list in the tests has a game the team lost or drew. A strong team can win every game.
Here lowest(new int[] {2, 1}) returns 0, a draw the team never played, so {2, 1} is a counterexample. low starts at 0, and when every score is above 0, no score is below low. Keep the counterexample as a test. After the repair it passes, and it stays as a regression test, the kind of check that Minimal reproductions ends with.
// The belief gives 1.
check("all wins", lowest(new int[] {2, 1}), 1);
// FAIL all wins: got 0, want 1Passing tests show that the belief holds for their inputs; one counterexample shows that it is false.
Precisely: letting a tool find counterexamples
Property-based testing states a belief as a property, such as “the result is one of the scores and no score is lower”. A tool then generates hundreds of inputs and looks for a counterexample. When it finds one, it shrinks the input, as in Minimal reproductions, before reporting it. For Java, jqwik is such a tool.
Experiment
Add a team that won every game
low starts at 0, not at a score from the list. A score replaces low only when it is below low. When every score is above 0, nothing replaces it, and lowest returns 0: a draw the team never played. Each of the three tests had a score at or below 0, where starting at 0 does no harm.
Why it matters Passing tests show that the belief holds for their inputs. To find a bug, try an input the code was not designed around, and keep the counterexample as a test after the repair.
All three tests print pass. Predict whether lowest is right for every list. Then add check("all wins", lowest(new int[] {2, 1}), 1); as the last line of main and run again.
class Main {
static int lowest(int[] scores) {
int low = 0;
for (int score : scores) {
if (score < low) {
low = score;
}
}
return low;
}
static void check(String name, int got, int want) {
if (got == want) {
System.out.println("pass " + name);
} else {
System.out.println(
"FAIL " + name + ": got " + got + ", want " + want);
}
}
public static void main(String[] args) {
check("mixed", lowest(new int[] {1, -1, 2}), -1);
check("a draw", lowest(new int[] {0, 3}), 0);
check("one game", lowest(new int[] {-2}), -2);
}
}
What to look for
Step into lowest for the first test. low is 0, and at score = 1 the stepper goes back to the for line without stopping on low = score: 1 is not below 0. After the edit, in the fourth call score is 2, then 1, low stays 0, and the stepper never stops on low = score. The check frame shows got = 0 and want = 1, and the last line of output is FAIL all wins: got 0, want 1.
Expected output
pass mixed
pass a draw
pass one game03 / Boundaries
Where do counterexamples usually hide?
Counterexamples are most often found at boundaries: inputs at the edge of what the code handles. Try these first: an empty input; a single element; equal values; the answer in the first place and in the last; zero; negative numbers, and no negative numbers at all. A loop that starts or stops one place too early, or a starting value such as 0, is wrong at a boundary and right everywhere else.
For each boundary, work out the expected value from the belief before you run it. The belief may say nothing about an input. This belief says nothing about an empty list. That is a finding too: decide what should happen, and add the decision to the belief.
The repair starts low at the first score, scores[0]. The table in this section runs the repaired lowest on one input for each boundary.
static int lowest(int[] scores) {
int low = scores[0];
// ... the loop as before
return low;
}Test the competing explanations
Predict first. These are recorded outcomes of the example, not tests running in your browser.
01One game: [3]
The belief gives 3.
Reveal the observation
Observed 3: pass.
What this rules in or out With one score, the start value is the answer, and the loop finds nothing lower.
02Equal scores: [1, 1]
The belief gives 1.
Reveal the observation
Observed 1: pass.
What this rules in or out A tie for the lowest score gives that score once.
03Lowest score last: [2, 0, -1]
The belief gives -1.
Reveal the observation
Observed -1: pass.
What this rules in or out The loop reaches the last score.
04Every game won: [2, 1]
The belief gives 1.
Reveal the observation
Observed 1: pass.
What this rules in or out The start value now comes from the list, not from 0.
05No games: an empty list
The belief covers lists with at least one score, so it predicts nothing.
Reveal the observation
Observed The program stops: java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0.
What this rules in or out Decide whether an empty list is a mistake in the calling code or an input to handle. The assertions section takes this up.
Try the edges first: empty, one, equal, first and last, zero, and negative.
04 / Tests that check nothing
Can a test pass without checking anything?
A test that cannot fail still prints pass. It tells you nothing. Three kinds are common.
The first kind has no comparison. It calls the function and prints the result. Nothing in it can fail.
The second kind gets its expected value from the code under test. It calls the same function again, or it repeats the same formula. So it compares the code with itself.
The third kind has an expected value copied from the output of one run. It checks that the output does not change. If that output was wrong, the test keeps the bug.
The rule from the first section prevents all three: the expected value comes from the belief, worked out without the code. To check a new test, break the code on purpose. For example, make the function return 0 at once, and run the test again. If the test still passes, it does not test that code.
The test below gets want from lowest itself. The experiment runs it on the counterexample from the earlier section.
static String testAllWins() {
int[] wins = {2, 1};
int want = lowest(wins);
int got = lowest(wins);
// got == want, whatever lowest returns
...
}A test must be able to fail: its expected value comes from the belief, not from the code.
Precisely: breaking the code on purpose
Mutation testing tools automate the check in this section. They make many small changes to the code, such as < to <=, run the tests after each, and report every change that no test noticed. For Java, PIT is such a tool.
Experiment
Take the expected value from the function itself
want and got are both results of lowest for the same list. Whatever lowest returns, the two are equal, so the comparison is always true. This test passes with the wrong lowest, and it would pass with a correct one. It cannot tell them apart.
Why it matters Work out the expected value from the belief, by hand. Then a wrong function gives a different value, and the test can fail.
You know that lowest for {2, 1} is wrong. Predict what this test prints. Then replace int want = lowest(wins); with int want = 1; and run again.
class Main {
static int lowest(int[] scores) {
int low = 0;
for (int score : scores) {
if (score < low) {
low = score;
}
}
return low;
}
static String testAllWins() {
int[] wins = {2, 1};
int want = lowest(wins);
int got = lowest(wins);
String verdict = "pass";
if (got != want) {
verdict = "FAIL";
}
return verdict + ": got " + got + ", want " + want;
}
public static void main(String[] args) {
System.out.println(testAllWins());
}
}
What to look for
Step into testAllWins. After its two calls to lowest, its frame shows wins = int[]@N [2, 1], want = 0, and got = 0, and the output is pass: got 0, want 0. int[] is the type, and @N is a number the stepper gives each array, so you can tell arrays apart. After the edit, want is 1, got is still 0, and the output is FAIL: got 0, want 1.
Expected output
pass: got 0, want 005 / Assertions
When should the program stop itself?
A test calls the code from outside, with inputs you chose. An assertion sits inside the code. It states that a condition is true at that point, every time the program reaches it. When the condition is false, the code itself is wrong, and the program should stop with a message.
Use assertions for programmer mistakes. A common one is a broken precondition: something that must be true when a function is called, such as “the list has at least one score”. The code that calls the function is responsible for it. Do not use assertions for problems that come from outside the program, such as a missing file, a score typed as text, or an empty list sent by another program. Those can happen when every line of the code is right. Handle them as errors and tell the user.
In Java, assert condition : message; throws an AssertionError when the condition is false, but only when the JVM was started with -ea, short for enable assertions. Without -ea, which is the default, the assert statement does nothing. A check that must always run is an if that throws an exception, such as IllegalArgumentException for a bad argument. Bad input from outside is an exception the caller is expected to catch.
static int lowest(int[] scores) {
assert scores.length > 0 : "lowest needs at least one score";
int low = scores[0];
// ... the loop as before
return low;
}Assert what only a programmer mistake can break, and remember that assert runs only with -ea. Throw an exception for everything else.
Coming from Python: Python’s assert runs unless Python starts with -O; Java’s assert is skipped unless the JVM starts with -ea, the opposite default.
Precisely: switching assertions on
java -ea switches assertions on for every class except the JDK’s own; -ea:com.example... switches them on for one package and its subpackages, and -da switches them off. An AssertionError is an Error, not an Exception, so a catch (Exception e) block does not catch it.
Experiment
Watch an assert that is switched off
Without -ea, the JVM skips every assert statement, so total > 0 is never evaluated. An if with throw is ordinary code, and it runs whatever options the JVM was started with.
Why it matters Java’s assert checks only while someone runs with -ea; Maven and Gradle switch it on when they run tests. A check that must hold in the running program is an if that throws.
Game 3’s length is 0, and the assert says a length is above 0. Predict the output. Then replace the assert line with if (total <= 0) throw new IllegalStateException("total must be above 0"); and run again.
class Main {
static int minutesLeft(int played, int total) {
assert total > 0 : "total must be above 0";
return total - played;
}
public static void main(String[] args) {
// each game's length in minutes; game 3's was left at 0
int[] totals = {40, 60, 0};
System.out.println(minutesLeft(30, totals[2]));
}
}
What to look for
The stepper stops on the assert line with total = 0, moves on to return total - played, and the output is -30: the stepper starts Java without -ea. After the edit, the stepper stops on the if line, and the run ends with java.lang.IllegalStateException: total must be above 0. Nothing is printed.
Expected output
-3006 / Try it
Can you find the counterexample?
Predict the output, then look for a counterexample before you reveal the answer. A team’s score sheet shows the top scorer of a game, by position in the team list. The belief is the comment at the top of the program.
class Example {
// topScorer should return the position of the highest score;
// on a tie, the earlier player's position.
static int topScorer(int[] points) {
int top = 0;
for (int i = 1; i < points.length; i++) {
if (points[i] >= points[top]) {
top = i;
}
}
return top;
}
static void check(String name, int got, int want) {
if (got == want) {
System.out.println("pass " + name);
} else {
System.out.println(
"FAIL " + name + ": got " + got + ", want " + want);
}
}
public static void main(String[] args) {
check("second", topScorer(new int[] {8, 12, 5}), 1);
check("first", topScorer(new int[] {20, 3}), 0);
check("alone", topScorer(new int[] {4}), 0);
}
}
What does this print, which input is a counterexample to the belief, and which boundary does it sit on?
Reveal the reasoning
It prints pass second, pass first, and pass alone. The belief is still false: topScorer(new int[] {9, 9}) returns 1 where the belief gives 0. The counterexample sits on the equal-values boundary: >= moves top to a later player with the same score. A tie below the top, such as {3, 3, 8}, would not fail, because 8 wins alone. The repair is > in place of >=, and the tie stays as a regression test.
One more boundary: for an empty team list, the function returns 0, the position of a player who does not exist. The belief needs a decision for that case too.
Practice tickets
- TEST-101 · Long games never finish on the score screen → Free
- TEST-102 · A relay team with one swimmer entered twice was accepted → Sign in · Pro
- TEST-103 · A screen that shows the wrong lift wait passes the checker → Sign in · Pro