Before you begin
Why this matters
A function can pass every check you wrote and still be wrong for an input you did not think of. Write down what the function should do before you check it. Then look for the input that proves it wrong, and you find the bug before someone else does.
What this helps you reason about
- Write a check whose expected value comes from what the function should do, not from what it does.
- Choose the inputs where code is most often wrong: empty, one, equal, first and last, zero, and negative.
- Tell a check that could fail from one that always passes.
A team app shows each team’s worst game of a season. The function behind it has three tests, and all three pass. This lesson asks what those passing tests prove, and how to find the input that proves the function wrong.
Already know this? The short version
- Write down what the function should return, and for which inputs, before you choose inputs; every expected value comes from that sentence, worked out by hand, never from running the code.
- Passing checks prove that sentence only for their inputs; one input that gives a wrong result disproves it, so try inputs the code was not designed around.
- Try the edges first: empty, one element, equal values, first and last place, zero, negative, and no negatives at all.
- A check with no comparison, or whose expected value comes from the code it checks, cannot fail.
- assert! panics in every build and debug_assert! only in debug builds; use them for programmer mistakes and return a Result for failures that come from outside.
Words in this lesson (7)
- test
- Code that calls a function with a chosen input, compares the result with an expected value worked out in advance, and reports pass or fail.
- counterexample
- One input for which a belief about a function is false. A single counterexample is enough to show that the belief is wrong.
- regression test
- A check, kept with the code, that fails while a bug is present and passes once it is fixed, so the bug cannot come back unnoticed.
- boundary
- An input at the edge of what a function handles, such as an empty list, a single element, equal values, the first or last element, zero, or a negative number.
- assertion
- A check written into the program that a condition is true at that point. assert!(condition, message) panics when the condition is false, in every build; debug_assert! does the same in debug builds only.
- precondition
- Something that must be true when a function is called, such as “the list is not empty”. The code that calls the function is responsible for it.
- panic
- Rust’s way of stopping because of a bug: the current thread stops with a message such as thread 'main' panicked at …. In a program with one thread, the program stops.
Examples are focused snippets; imports, enclosing functions, or classes may be omitted. C examples use C11; Haskell examples use Haskell 2010. Expand the notes for language-specific depth.
01 / The belief
What do you believe the function does?
A test checks a belief about a function, so write the belief down first. The belief is one sentence that says what the function returns, and for which inputs. Then write the tests. A test is a call with a chosen input, plus the value that the belief says the call should return, called the expected value. Work out each expected value from the belief, by hand, before you run the code. The test then compares two things made separately: what the code returns and what the belief says.
For each game, the app stores a score: the team’s points minus the other team’s points. So -1 means the team lost by one point, 0 means a draw, and 2 means the team won by two points. The belief: lowest returns the smallest score in a list that has at least one score. The tests use a small function, check(name, got, want), where got is the value lowest returned and want is the expected value. It prints pass when got equals want, and FAIL with both values when they differ. Its full code is in the experiment of the next section.
for &score in scores visits each score in turn. The loop gives a reference to each score, a &i32. The pattern &score matches that reference and copies the i32 it refers to into score, so score is a plain number. The copy is allowed because i32 is Copy.
// lowest should return the smallest score of a list
// that has at least one score.
fn lowest(scores: &[i32]) -> i32 {
let mut low = 0;
for &score in scores {
if score < low {
low = score;
}
}
low
}
// Each line prints pass.
check("mixed", lowest(&[1, -1, 2]), -1);
check("a draw", lowest(&[0, 3]), 0);
check("one game", lowest(&[-2]), -2);Write down your belief and each expected value before you run the code.
Coming from Python: pytest’s assert got == want and Rust’s assert_eq!(got, want) are the same test: a call plus an expected value, with both values reported on failure.
Precisely: in a real project
A Rust project keeps unit tests in a tests module next to the code. #[cfg(test)] leaves the module out of normal builds, and use super::* brings lowest into it. #[test] marks a function as a test, and cargo test runs every such function. assert_eq!(got, want) panics with both values when they differ, and a panic fails that test only; the others still run.
This lesson uses a plain check function instead, because the stepper runs one file with no test tool.
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn all_wins() {
assert_eq!(lowest(&[2, 1]), 1);
}
}02 / Counterexamples
Which input would prove the belief wrong?
Passing tests show that the belief holds for their inputs. They say nothing about other inputs. To learn more, look for a counterexample: one input for which the belief is false. One counterexample is enough to show that a belief is wrong, however many tests pass.
Look first at inputs the code was not designed around. Every list in the tests has a game the team lost or drew. A strong team can win every game.
Here lowest(&[2, 1]) returns 0, a draw the team never played, so [2, 1] is a counterexample. low starts at 0, and when every score is above 0, no score is below low. Keep the counterexample as a test. After the repair it passes, and it stays as a regression test, the kind of check that Minimal reproductions ends with.
// The belief gives 1.
check("all wins", lowest(&[2, 1]), 1);
// FAIL all wins: got 0, want 1Passing tests show that the belief holds for their inputs; one counterexample shows that it is false.
Precisely: letting a tool find counterexamples
Property-based testing states a belief as a property, such as “the result is one of the scores and no score is lower”. A tool then generates hundreds of inputs and looks for a counterexample. When it finds one, it shrinks the input, as in Minimal reproductions, before reporting it. For Rust, proptest and quickcheck are such tools.
Experiment
Add a team that won every game
low starts at 0, not at a score from the list. A score replaces low only when it is below low. When every score is above 0, nothing replaces it, and lowest returns 0: a draw the team never played. Each of the three tests had a score at or below 0, where starting at 0 does no harm.
Why it matters Passing tests show that the belief holds for their inputs. To find a bug, try an input the code was not designed around, and keep the counterexample as a test after the repair.
All three tests print pass. Predict whether lowest is right for every list. Then add check("all wins", lowest(&[2, 1]), 1); as the last line of main and run again.
fn lowest(scores: &[i32]) -> i32 {
let mut low = 0;
for &score in scores {
if score < low {
low = score;
}
}
low
}
fn check(name: &str, got: i32, want: i32) {
if got == want {
println!("pass {}", name);
} else {
println!("FAIL {}: got {}, want {}", name, got, want);
}
}
fn main() {
check("mixed", lowest(&[1, -1, 2]), -1);
check("a draw", lowest(&[0, 3]), 0);
check("one game", lowest(&[-2]), -2);
}
What to look for
Step into lowest for the first test. scores shows as [1, -1, 2] (len=3, data=0x…): the scores, how many there are, and the address where they start. low is 0, and at score = 1 it stays 0. After the edit, in the fourth call score is 2, then 1, low stays 0, and the stepper never stops on low = score. The check frame shows got = 0, want = 1, and name = "all wins" (len=8, data=0x…). The last line of output is FAIL all wins: got 0, want 1.
Expected output
pass mixed
pass a draw
pass one game03 / Boundaries
Where do counterexamples usually hide?
Counterexamples are most often found at boundaries: inputs at the edge of what the code handles. Try these first: an empty input; a single element; equal values; the answer in the first place and in the last; zero; negative numbers, and no negative numbers at all. A loop that starts or stops one place too early, or a starting value such as 0, is wrong at a boundary and right everywhere else.
For each boundary, work out the expected value from the belief before you run it. The belief may say nothing about an input. This belief says nothing about an empty list. That is a finding too: decide what should happen, and add the decision to the belief.
The repair starts low at the first score, scores[0]. The table in this section runs the repaired lowest on one input for each boundary.
fn lowest(scores: &[i32]) -> i32 {
let mut low = scores[0];
// ... the loop as before
low
}Test the competing explanations
Predict first. These are recorded outcomes of the example, not tests running in your browser.
01One game: [3]
The belief gives 3.
Reveal the observation
Observed 3: pass.
What this rules in or out With one score, the start value is the answer, and the loop finds nothing lower.
02Equal scores: [1, 1]
The belief gives 1.
Reveal the observation
Observed 1: pass.
What this rules in or out A tie for the lowest score gives that score once.
03Lowest score last: [2, 0, -1]
The belief gives -1.
Reveal the observation
Observed -1: pass.
What this rules in or out The loop reaches the last score.
04Every game won: [2, 1]
The belief gives 1.
Reveal the observation
Observed 1: pass.
What this rules in or out The start value now comes from the list, not from 0.
05No games: an empty list
The belief covers lists with at least one score, so it predicts nothing.
Reveal the observation
Observed The program stops: thread 'main' panicked at 'index out of bounds: the len is 0 but the index is 0'.
What this rules in or out Decide whether an empty list is a mistake in the calling code or an input to handle. The assertions section takes this up.
Try the edges first: empty, one, equal, first and last, zero, and negative.
04 / Tests that check nothing
Can a test pass without checking anything?
A test that cannot fail still prints pass. It tells you nothing. Three kinds are common.
The first kind has no comparison. It calls the function and prints the result. Nothing in it can fail.
The second kind gets its expected value from the code under test. It calls the same function again, or it repeats the same formula. So it compares the code with itself.
The third kind has an expected value copied from the output of one run. It checks that the output does not change. If that output was wrong, the test keeps the bug.
The rule from the first section prevents all three: the expected value comes from the belief, worked out without the code. To check a new test, break the code on purpose. For example, make the function return 0 at once, and run the test again. If the test still passes, it does not test that code.
The test below gets want from lowest itself. The experiment runs it on the counterexample from the earlier section.
fn test_all_wins() -> String {
let wins = [2, 1];
let want = lowest(&wins);
let got = lowest(&wins);
// got == want, whatever lowest returns
...
}A test must be able to fail: its expected value comes from the belief, not from the code.
Precisely: breaking the code on purpose
Mutation testing tools automate the check in this section. They make many small changes to the code, such as < to <=, run the tests after each, and report every change that no test noticed. For Rust, cargo-mutants is such a tool.
Experiment
Take the expected value from the function itself
want and got are both results of lowest for the same list. Whatever lowest returns, the two are equal, so the comparison is always true. This test passes with the wrong lowest, and it would pass with a correct one. It cannot tell them apart.
Why it matters Work out the expected value from the belief, by hand. Then a wrong function gives a different value, and the test can fail.
You know that lowest(&[2, 1]) is wrong. Predict what this test prints. Then replace let want = lowest(&wins); with let want = 1; and run again.
fn lowest(scores: &[i32]) -> i32 {
let mut low = 0;
for &score in scores {
if score < low {
low = score;
}
}
low
}
fn test_all_wins() -> String {
let wins = [2, 1];
let want = lowest(&wins);
let got = lowest(&wins);
let mut verdict = "pass";
if got != want {
verdict = "FAIL";
}
format!("{}: got {}, want {}", verdict, got, want)
}
fn main() {
println!("{}", test_all_wins());
}
What to look for
Step into test_all_wins. After its two calls to lowest, its frame shows wins = [2, 1], want = 0, and got = 0, and the output is pass: got 0, want 0. After the edit, want is 1, got is still 0, and the output is FAIL: got 0, want 1.
Expected output
pass: got 0, want 005 / Assertions
When should the program stop itself?
A test calls the code from outside, with inputs you chose. An assertion sits inside the code. It states that a condition is true at that point, every time the program reaches it. When the condition is false, the code itself is wrong, and the program should stop with a message.
Use assertions for programmer mistakes. A common one is a broken precondition: something that must be true when a function is called, such as “the list has at least one score”. The code that calls the function is responsible for it. Do not use assertions for problems that come from outside the program, such as a missing file, a score typed as text, or an empty list sent by another program. Those can happen when every line of the code is right. Handle them as errors and tell the user.
In Rust, assert!(condition, message) panics when the condition is false: the thread stops with the message, and in a program with one thread the program stops. assert! stays in every build. debug_assert! is the same check, left out of release builds such as cargo build --release. For failures that come from outside the program, return a Result and let the caller decide.
fn lowest(scores: &[i32]) -> i32 {
assert!(!scores.is_empty(), "lowest needs at least one score");
let mut low = scores[0];
// ... the loop as before
low
}Use assert! for what only a programmer mistake can break; it stays in every build, and debug_assert! does not. Return a Result for everything else.
Coming from C: C’s NDEBUG removes every assert; Rust’s assert! stays in release builds, and only debug_assert! is left out.
Precisely: what decides debug_assert!
debug_assert! is compiled in when debug assertions are on: by default in cargo’s dev profile and off in its release profile, and controlled directly by rustc’s -C debug-assertions. Indexing past the end of a slice panics in every build, which is why the empty list already stops the program; the assert! gives the panic a message about the caller’s mistake.
Experiment
Send a length of 0 to assert!
assert! checks its condition every time the line runs and panics when the condition is false. This program has one thread, so the panic ends the program before println! runs.
Why it matters assert! is part of every build. debug_assert! is the same check for debug builds only; use it for a check too slow to keep, and remember that a release build will not run it.
The program asks for game 2, whose length is 60. Predict the output. Then change totals[1] to totals[2], game 3, whose length was left at 0, and run again.
fn minutes_left(played: i32, total: i32) -> i32 {
assert!(total > 0, "total must be above 0");
total - played
}
fn main() {
// each game's length in minutes; game 3's was left at 0 by mistake
let totals = [40, 60, 0];
println!("{}", minutes_left(30, totals[1]));
}
What to look for
With totals[1], minutes_left has total = 60. The stepper stops on the assert! line, then on total - played, and the output is 30. After the edit, total = 0, and the run stops at the assert! line with thread 'main' panicked at 'total must be above 0', main.rs:2:5. Nothing is printed.
Expected output
3006 / Try it
Can you find the counterexample?
Predict the output, then look for a counterexample before you reveal the answer. A team’s score sheet shows the top scorer of a game, by position in the team list. The belief is the comment at the top of the program. 1..points.len() counts from 1 up to, but not including, the length, and usize is Rust’s type for positions.
// top_scorer should return the position of the highest score;
// on a tie, the earlier player's position.
fn top_scorer(points: &[i32]) -> usize {
let mut top = 0;
for i in 1..points.len() {
if points[i] >= points[top] {
top = i;
}
}
top
}
fn check(name: &str, got: usize, want: usize) {
if got == want {
println!("pass {}", name);
} else {
println!("FAIL {}: got {}, want {}", name, got, want);
}
}
fn main() {
check("second", top_scorer(&[8, 12, 5]), 1);
check("first", top_scorer(&[20, 3]), 0);
check("alone", top_scorer(&[4]), 0);
}
What does this print, which input is a counterexample to the belief, and which boundary does it sit on?
Reveal the reasoning
It prints pass second, pass first, and pass alone. The belief is still false: top_scorer(&[9, 9]) returns 1 where the belief gives 0. The counterexample sits on the equal-values boundary: >= moves top to a later player with the same score. A tie below the top, such as [3, 3, 8], would not fail, because 8 wins alone. The repair is > in place of >=, and the tie stays as a regression test. The library shortcut has the same trap: iter().enumerate().max_by_key gives the last of equal items.
One more boundary: for an empty team list, the function returns 0, the position of a player who does not exist. The belief needs a decision for that case too.
Practice tickets
- TEST-101 · Long games never finish on the score screen → Free
- TEST-102 · A relay team with one swimmer entered twice was accepted → Sign in · Pro
- TEST-103 · A screen that shows the wrong lift wait passes the checker → Sign in · Pro