Before you begin
Why this matters
A function can pass every check you wrote and still be wrong for an input you did not think of. Write down what the function should do before you check it. Then look for the input that proves it wrong, and you find the bug before someone else does.
What this helps you reason about
- Write a check whose expected value comes from what the function should do, not from what it does.
- Choose the inputs where code is most often wrong: empty, one, equal, first and last, zero, and negative.
- Tell a check that could fail from one that always passes.
A team app shows each team’s worst game of a season. The function behind it has three tests, and all three pass. This lesson asks what those passing tests prove, and how to find the input that proves the function wrong.
Already know this? The short version
- Write down what the function should return, and for which inputs, before you choose inputs; every expected value comes from that sentence, worked out by hand, never from running the code.
- Passing checks prove that sentence only for their inputs; one input that gives a wrong result disproves it, so try inputs the code was not designed around.
- Try the edges first: empty, one element, equal values, first and last place, zero, negative, and no negatives at all.
- A check with no comparison, or whose expected value comes from the code it checks, cannot fail.
- Go has no assert statement: a programmer mistake is an if that calls panic, which no build setting removes, and a failure the caller should handle is returned as an error.
Words in this lesson (7)
- test
- Code that calls a function with a chosen input, compares the result with an expected value worked out in advance, and reports pass or fail.
- counterexample
- One input for which a belief about a function is false. A single counterexample is enough to show that the belief is wrong.
- regression test
- A check, kept with the code, that fails while a bug is present and passes once it is fixed, so the bug cannot come back unnoticed.
- boundary
- An input at the edge of what a function handles, such as an empty list, a single element, equal values, the first or last element, zero, or a negative number.
- assertion
- A check written into the program that a condition is true at that point. Go has no assert statement; Go code writes an if that calls panic when the condition is false.
- precondition
- Something that must be true when a function is called, such as “the list is not empty”. The code that calls the function is responsible for it.
- panic
- Go’s way of stopping a program because of a mistake in the code. panic("message") stops the current function and every function that called it, running their deferred calls on the way. Unless a deferred call recovers, the program prints panic: message and a stack trace, then exits.
Examples are focused snippets; imports, enclosing functions, or classes may be omitted. C examples use C11; Haskell examples use Haskell 2010. Expand the notes for language-specific depth.
01 / The belief
What do you believe the function does?
A test checks a belief about a function, so write the belief down first. The belief is one sentence that says what the function returns, and for which inputs. Then write the tests. A test is a call with a chosen input, plus the value that the belief says the call should return, called the expected value. Work out each expected value from the belief, by hand, before you run the code. The test then compares two things made separately: what the code returns and what the belief says.
For each game, the app stores a score: the team’s points minus the other team’s points. So -1 means the team lost by one point, 0 means a draw, and 2 means the team won by two points. The belief: lowest returns the smallest score in a list that has at least one score. The tests use a small function, check(name, got, want), where got is the value lowest returned and want is the expected value. It prints pass when got equals want, and FAIL with both values when they differ. Its full code is in the experiment of the next section. for _, score := range scores visits each score in turn; the _ throws away its position.
// lowest should return the smallest score of a list
// that has at least one score.
func lowest(scores []int) int {
low := 0
for _, score := range scores {
if score < low {
low = score
}
}
return low
}
// Each line prints pass.
check("mixed", lowest([]int{1, -1, 2}), -1)
check("a draw", lowest([]int{0, 3}), 0)
check("one game", lowest([]int{-2}), -2)Write down your belief and each expected value before you run the code.
Coming from Java: Go’s standard library has no assertEquals: a Go test is an if that compares got with want and calls t.Errorf, the same shape as check here.
Precisely: in a real project
go test runs every function named TestXxx in files that end in _test.go. A Go test is usually table-driven: a slice of cases, each with a name, an input, and want, and a loop that runs each case. t.Run gives each case its own name in the report. t.Errorf marks the test as failed and goes on; t.Fatalf also stops that test. A helper that calls t.Errorf can call t.Helper first, so the report names the caller’s line.
This lesson uses a plain check function instead, because the stepper runs one file with no test tool.
// lowest_test.go; run with: go test
package main
import "testing"
func TestLowest(t *testing.T) {
cases := []struct {
name string
scores []int
want int
}{
{"mixed", []int{1, -1, 2}, -1},
{"a draw", []int{0, 3}, 0},
{"all wins", []int{2, 1}, 1},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
got := lowest(c.scores)
if got != c.want {
t.Errorf("got %d, want %d", got, c.want)
}
})
}
}02 / Counterexamples
Which input would prove the belief wrong?
Passing tests show that the belief holds for their inputs. They say nothing about other inputs. To learn more, look for a counterexample: one input for which the belief is false. One counterexample is enough to show that a belief is wrong, however many tests pass.
Look first at inputs the code was not designed around. Every list in the tests has a game the team lost or drew. A strong team can win every game.
Here lowest([]int{2, 1}) returns 0, a draw the team never played, so [2, 1] is a counterexample. low starts at 0, and when every score is above 0, no score is below low. Keep the counterexample as a test. After the repair it passes, and it stays as a regression test, the kind of check that Minimal reproductions ends with.
// The belief gives 1.
check("all wins", lowest([]int{2, 1}), 1)
// FAIL all wins got 0 want 1Passing tests show that the belief holds for their inputs; one counterexample shows that it is false.
Precisely: letting a tool find counterexamples
Property-based testing states a belief as a property, such as “the result is one of the scores and no score is lower”. A tool then generates hundreds of inputs and looks for a counterexample. When it finds one, it shrinks the input, as in Minimal reproductions, before reporting it. Go has fuzzing built into go test since Go 1.18: a FuzzXxx function receives generated inputs, and a failing input is saved as a new test case.
Experiment
Add a team that won every game
low starts at 0, not at a score from the list. A score replaces low only when it is below low. When every score is above 0, nothing replaces it, and lowest returns 0: a draw the team never played. Each of the three tests had a score at or below 0, where starting at 0 does no harm.
Why it matters Passing tests show that the belief holds for their inputs. To find a bug, try an input the code was not designed around, and keep the counterexample as a test after the repair.
All three tests print pass. Predict whether lowest is right for every list. Then add check("all wins", lowest([]int{2, 1}), 1) as the last line of main and run again.
package main
import "fmt"
func lowest(scores []int) int {
low := 0
for _, score := range scores {
if score < low {
low = score
}
}
return low
}
func check(name string, got int, want int) {
if got == want {
fmt.Println("pass", name)
} else {
fmt.Println("FAIL", name, "got", got, "want", want)
}
}
func main() {
check("mixed", lowest([]int{1, -1, 2}), -1)
check("a draw", lowest([]int{0, 3}), 0)
check("one game", lowest([]int{-2}), -2)
}
What to look for
Step into lowest for the first test. low is 0. The stepper stops on the if line with score = 1, then moves to score = -1 without running low = score: 1 is not below 0. After the edit, in the fourth call score is 2, then 1, low stays 0, and the stepper never stops on low = score. The check frame shows got = 0 and want = 1, and the last line of output is FAIL all wins got 0 want 1.
Expected output
pass mixed
pass a draw
pass one game03 / Boundaries
Where do counterexamples usually hide?
Counterexamples are most often found at boundaries: inputs at the edge of what the code handles. Try these first: an empty input; a single element; equal values; the answer in the first place and in the last; zero; negative numbers, and no negative numbers at all. A loop that starts or stops one place too early, or a starting value such as 0, is wrong at a boundary and right everywhere else.
For each boundary, work out the expected value from the belief before you run it. The belief may say nothing about an input. This belief says nothing about an empty list. That is a finding too: decide what should happen, and add the decision to the belief.
The repair starts low at the first score, scores[0]. The table in this section runs the repaired lowest on one input for each boundary.
func lowest(scores []int) int {
low := scores[0]
// ... the loop as before
return low
}Test the competing explanations
Predict first. These are recorded outcomes of the example, not tests running in your browser.
01One game: [3]
The belief gives 3.
Reveal the observation
Observed 3: pass.
What this rules in or out With one score, the start value is the answer, and the loop finds nothing lower.
02Equal scores: [1, 1]
The belief gives 1.
Reveal the observation
Observed 1: pass.
What this rules in or out A tie for the lowest score gives that score once.
03Lowest score last: [2, 0, -1]
The belief gives -1.
Reveal the observation
Observed -1: pass.
What this rules in or out The loop reaches the last score.
04Every game won: [2, 1]
The belief gives 1.
Reveal the observation
Observed 1: pass.
What this rules in or out The start value now comes from the list, not from 0.
05No games: an empty list
The belief covers lists with at least one score, so it predicts nothing.
Reveal the observation
Observed The program stops: panic: runtime error: index out of range [0] with length 0.
What this rules in or out Decide whether an empty list is a mistake in the calling code or an input to handle. The assertions section takes this up.
Try the edges first: empty, one, equal, first and last, zero, and negative.
04 / Tests that check nothing
Can a test pass without checking anything?
A test that cannot fail still prints pass. It tells you nothing. Three kinds are common.
The first kind has no comparison. It calls the function and prints the result. Nothing in it can fail.
The second kind gets its expected value from the code under test. It calls the same function again, or it repeats the same formula. So it compares the code with itself.
The third kind has an expected value copied from the output of one run. It checks that the output does not change. If that output was wrong, the test keeps the bug.
The rule from the first section prevents all three: the expected value comes from the belief, worked out without the code. To check a new test, break the code on purpose. For example, make the function return 0 at once, and run the test again. If the test still passes, it does not test that code.
The test below gets want from lowest itself. The experiment runs it on the counterexample from the earlier section. fmt.Sprint joins its arguments into one string.
func testAllWins() string {
wins := []int{2, 1}
want := lowest(wins)
got := lowest(wins)
// got == want, whatever lowest returns
...
}A test must be able to fail: its expected value comes from the belief, not from the code.
Precisely: breaking the code on purpose
Mutation testing tools automate the check in this section. They make many small changes to the code, such as < to <=, run the tests after each, and report every change that no test noticed. For Go, go-mutesting is such a tool.
Experiment
Take the expected value from the function itself
want and got are both results of lowest for the same list. Whatever lowest returns, the two are equal, so the comparison is always true. This test passes with the wrong lowest, and it would pass with a correct one. It cannot tell them apart.
Why it matters Work out the expected value from the belief, by hand. Then a wrong function gives a different value, and the test can fail.
You know that lowest([]int{2, 1}) is wrong. Predict what this test prints. Then replace want := lowest(wins) with want := 1 and run again.
package main
import "fmt"
func lowest(scores []int) int {
low := 0
for _, score := range scores {
if score < low {
low = score
}
}
return low
}
func testAllWins() string {
wins := []int{2, 1}
want := lowest(wins)
got := lowest(wins)
verdict := "pass"
if got != want {
verdict = "FAIL"
}
return fmt.Sprint(verdict, ": got ", got, ", want ", want)
}
func main() {
fmt.Println(testAllWins())
}
What to look for
Step into testAllWins. After its two calls to lowest, its frame shows want = 0 and got = 0, and the output is pass: got 0, want 0. After the edit, want is 1, got is still 0, and the output is FAIL: got 0, want 1.
Expected output
pass: got 0, want 005 / Assertions
When should the program stop itself?
A test calls the code from outside, with inputs you chose. An assertion sits inside the code. It states that a condition is true at that point, every time the program reaches it. When the condition is false, the code itself is wrong, and the program should stop with a message.
Use assertions for programmer mistakes. A common one is a broken precondition: something that must be true when a function is called, such as “the list has at least one score”. The code that calls the function is responsible for it. Do not use assertions for problems that come from outside the program, such as a missing file, a score typed as text, or an empty list sent by another program. Those can happen when every line of the code is right. Handle them as errors and tell the user.
Go has no assert statement on purpose. The Go FAQ says that programmers too often use assertions to avoid thinking about proper error handling. For a programmer mistake, Go code checks the condition with if and calls panic. panic("message") stops the current function. Its deferred calls run first; a deferred call is one that defer scheduled to run when the function ends. Then each calling function stops in the same way. When no deferred call recovers, the program prints panic: message and a stack trace, the list of calls that were running, and exits with status 2. No build setting removes a panic. A failure the caller should handle, such as bad input, is returned as an error value instead.
func lowest(scores []int) int {
if len(scores) == 0 {
panic("lowest needs at least one score")
}
low := scores[0]
// ... the loop as before
return low
}Panic for what only a programmer mistake can break; no build setting removes the check. Return an error for everything else.
Coming from Python: Python’s assert statement has no Go equivalent; Go code writes if with panic, and no flag switches the panic off.
Precisely: recover
A deferred function can call recover to stop a panic and let the program go on; the net/http server does this for each request. That makes a panic catchable, but Go code still uses it for mistakes in the code, not as a way to report ordinary failures.
Experiment
Send a length of 0 to the panic
panic stops minutesLeft and every function that called it. Nothing recovers from it here, so the program ends, and the subtraction with a length of 0 never runs.
Why it matters Go has no switch that removes a panic. A panic is for a mistake in the code; a problem the caller should handle is returned as an error value.
The program asks for game 2, whose length is 60. Predict the output. Then change totals[1] to totals[2], game 3, whose length was left at 0, and run again.
package main
import "fmt"
func minutesLeft(played int, total int) int {
if total <= 0 {
panic("total must be above 0")
}
return total - played
}
func main() {
// each game's length in minutes; game 3's was left at 0 by mistake
totals := []int{40, 60, 0}
fmt.Println(minutesLeft(30, totals[1]))
}
What to look for
With totals[1], minutesLeft has total = 60. The stepper stops on the if line, then on return total - played, and the output is 30. After the edit, total = 0, the stepper stops on the panic line, and the run stops with the message "total must be above 0". Nothing is printed: the panic ended the program before Println. Outside the stepper, the program prints panic: total must be above 0, then a stack trace, and exits with status 2.
Expected output
3006 / Try it
Can you find the counterexample?
Predict the output, then look for a counterexample before you reveal the answer. A team’s score sheet shows the top scorer of a game, by position in the team list. The belief is the comment at the top of the program.
package main
import "fmt"
// topScorer should return the position of the highest score;
// on a tie, the earlier player's position.
func topScorer(points []int) int {
top := 0
for i := 1; i < len(points); i++ {
if points[i] >= points[top] {
top = i
}
}
return top
}
func check(name string, got int, want int) {
if got == want {
fmt.Println("pass", name)
} else {
fmt.Println("FAIL", name, "got", got, "want", want)
}
}
func main() {
check("second", topScorer([]int{8, 12, 5}), 1)
check("first", topScorer([]int{20, 3}), 0)
check("alone", topScorer([]int{4}), 0)
}
What does this print, which input is a counterexample to the belief, and which boundary does it sit on?
Reveal the reasoning
It prints pass second, pass first, and pass alone. The belief is still false: topScorer([]int{9, 9}) returns 1 where the belief gives 0. The counterexample sits on the equal-values boundary: >= moves top to a later player with the same score. A tie below the top, such as [3, 3, 8], would not fail, because 8 wins alone. The repair is > in place of >=, and the tie stays as a regression test.
One more boundary: for an empty team list, the function returns 0, the position of a player who does not exist. The belief needs a decision for that case too.
Practice tickets
- TEST-101 · Long games never finish on the score screen → Free
- TEST-102 · A relay team with one swimmer entered twice was accepted → Sign in · Pro
- TEST-103 · A screen that shows the wrong lift wait passes the checker → Sign in · Pro