Before you begin
Why this matters
A function can pass every check you wrote and still be wrong for an input you did not think of. Write down what the function should do before you check it. Then look for the input that proves it wrong, and you find the bug before someone else does.
What this helps you reason about
- Write a check whose expected value comes from what the function should do, not from what it does.
- Choose the inputs where code is most often wrong: empty, one, equal, first and last, zero, and negative.
- Tell a check that could fail from one that always passes.
A team app shows each team’s worst game of a season. The function behind it has three tests, and all three pass. This lesson asks what those passing tests prove, and how to find the input that proves the function wrong.
Already know this? The short version
- Write down what the function should return, and for which inputs, before you choose inputs; every expected value comes from that sentence, worked out by hand, never from running the code.
- Passing checks prove that sentence only for their inputs; one input that gives a wrong result disproves it, so try inputs the code was not designed around.
- Try the edges first: empty, one element, equal values, first and last place, zero, negative, and no negatives at all.
- A check with no comparison, or whose expected value comes from the code it checks, cannot fail.
- console.assert prints a message and the program continues; throw an Error for a programmer mistake that must stop the program, and handle bad input as an error the caller expects.
Words in this lesson (7)
- test
- Code that calls a function with a chosen input, compares the result with an expected value worked out in advance, and reports pass or fail.
- counterexample
- One input for which a belief about a function is false. A single counterexample is enough to show that the belief is wrong.
- regression test
- A check, kept with the code, that fails while a bug is present and passes once it is fixed, so the bug cannot come back unnoticed.
- boundary
- An input at the edge of what a function handles, such as an empty list, a single element, equal values, the first or last element, zero, or a negative number.
- standard output and standard error
- The two output streams every program starts with. Standard output is for normal results; standard error is for messages about problems.
- assertion
- A check written into the program that a condition is true at that point. JavaScript has no assert statement: console.assert only prints a message when the condition is false, and a check that must stop the program throws an Error.
- precondition
- Something that must be true when a function is called, such as “the list is not empty”. The code that calls the function is responsible for it.
Examples are focused snippets; imports, enclosing functions, or classes may be omitted. C examples use C11; Haskell examples use Haskell 2010. Expand the notes for language-specific depth.
01 / The belief
What do you believe the function does?
A test checks a belief about a function, so write the belief down first. The belief is one sentence that says what the function returns, and for which inputs. Then write the tests. A test is a call with a chosen input, plus the value that the belief says the call should return, called the expected value. Work out each expected value from the belief, by hand, before you run the code. The test then compares two things made separately: what the code returns and what the belief says.
For each game, the app stores a score: the team’s points minus the other team’s points. So -1 means the team lost by one point, 0 means a draw, and 2 means the team won by two points. The belief: lowest returns the smallest score in a list that has at least one score. The tests use a small function, check(name, got, want), where got is the value lowest returned and want is the expected value. It prints pass when got equals want, and FAIL with both values when they differ. Its full code is in the experiment of the next section. for (const score of scores) visits each score in turn.
// lowest should return the smallest score of a list
// that has at least one score.
function lowest(scores: number[]): number {
let low = 0;
for (const score of scores) {
if (score < low) {
low = score;
}
}
return low;
}
// Each line prints pass.
check("mixed", lowest([1, -1, 2]), -1);
check("a draw", lowest([0, 3]), 0);
check("one game", lowest([-2]), -2);Write down your belief and each expected value before you run the code.
Coming from the Jest test library: expect(lowest([2, 1])).toBe(1) is the same test as check("all wins", lowest([2, 1]), 1): a call plus an expected value.
Precisely: in a real project
A real project keeps tests in their own file, and a test runner finds and runs them. Node.js has one built in, node:test, with assertions from node:assert. Vitest and Jest are common alternatives that also run TypeScript files directly. Below, test gives the test a name and a function to run; () => { … } is a function with no name and no parameters. assert.equal throws when the two values differ, and the runner reports that test as failed and goes on to the next one.
This lesson uses a plain check function instead, because the stepper runs one file with no test tool.
import { test } from "node:test";
import assert from "node:assert/strict";
import { lowest } from "./lowest.ts";
test("all wins", () => {
assert.equal(lowest([2, 1]), 1);
});02 / Counterexamples
Which input would prove the belief wrong?
Passing tests show that the belief holds for their inputs. They say nothing about other inputs. To learn more, look for a counterexample: one input for which the belief is false. One counterexample is enough to show that a belief is wrong, however many tests pass.
Look first at inputs the code was not designed around. Every list in the tests has a game the team lost or drew. A strong team can win every game.
Here lowest([2, 1]) returns 0, a draw the team never played, so [2, 1] is a counterexample. low starts at 0, and when every score is above 0, no score is below low. Keep the counterexample as a test. After the repair it passes, and it stays as a regression test, the kind of check that Minimal reproductions ends with.
// The belief gives 1.
check("all wins", lowest([2, 1]), 1);
// FAIL all wins: got 0, want 1Passing tests show that the belief holds for their inputs; one counterexample shows that it is false.
Precisely: letting a tool find counterexamples
Property-based testing states a belief as a property, such as “the result is one of the scores and no score is lower”. A tool then generates hundreds of inputs and looks for a counterexample. When it finds one, it shrinks the input, as in Minimal reproductions, before reporting it. For TypeScript, fast-check is such a tool; this lesson does the same search by hand.
Experiment
Add a team that won every game
low starts at 0, not at a score from the list. A score replaces low only when it is below low. When every score is above 0, nothing replaces it, and lowest returns 0: a draw the team never played. Each of the three tests had a score at or below 0, where starting at 0 does no harm.
Why it matters Passing tests show that the belief holds for their inputs. To find a bug, try an input the code was not designed around, and keep the counterexample as a test after the repair.
All three tests print pass. Predict whether lowest is right for every list. Then add check("all wins", lowest([2, 1]), 1); as the last line and run again.
function lowest(scores: number[]): number {
let low = 0;
for (const score of scores) {
if (score < low) {
low = score;
}
}
return low;
}
function check(name: string, got: number, want: number): void {
if (got === want) {
console.log("pass " + name);
} else {
console.log("FAIL " + name + ": got " + got + ", want " + want);
}
}
check("mixed", lowest([1, -1, 2]), -1);
check("a draw", lowest([0, 3]), 0);
check("one game", lowest([-2]), -2);
What to look for
Step into lowest for the first test, [1, -1, 2]. Before the loop low is 0, and at score = 1 it stays 0, because 1 is not below 0; the test still passes because -1 comes next. After the edit, watch the fourth call: score is 2, then 1, low stays 0, and the stepper never stops on low = score. The check frame shows got = 0 and want = 1, and the last line of output is FAIL all wins: got 0, want 1. The other three still pass. The top-level frame lists lowest as function lowest(scores) and check as function check(name, got, want): each is a variable that holds a function. The stepper can stop on the same line more than once; that is normal for this debugger.
Expected output
pass mixed
pass a draw
pass one game03 / Boundaries
Where do counterexamples usually hide?
Counterexamples are most often found at boundaries: inputs at the edge of what the code handles. Try these first: an empty input; a single element; equal values; the answer in the first place and in the last; zero; negative numbers, and no negative numbers at all. A loop that starts or stops one place too early, or a starting value such as 0, is wrong at a boundary and right everywhere else.
For each boundary, work out the expected value from the belief before you run it. The belief may say nothing about an input. This belief says nothing about an empty list. That is a finding too: decide what should happen, and add the decision to the belief.
The repair starts low at the first score, scores[0]. The table in this section runs the repaired lowest on one input for each boundary.
function lowest(scores: number[]): number {
let low = scores[0];
// ... the loop as before
return low;
}Test the competing explanations
Predict first. These are recorded outcomes of the example, not tests running in your browser.
01One game: [3]
The belief gives 3.
Reveal the observation
Observed 3: pass.
What this rules in or out With one score, the start value is the answer, and the loop finds nothing lower.
02Equal scores: [1, 1]
The belief gives 1.
Reveal the observation
Observed 1: pass.
What this rules in or out A tie for the lowest score gives that score once.
03Lowest score last: [2, 0, -1]
The belief gives -1.
Reveal the observation
Observed -1: pass.
What this rules in or out The loop reaches the last score.
04Every game won: [2, 1]
The belief gives 1.
Reveal the observation
Observed 1: pass.
What this rules in or out The start value now comes from the list, not from 0.
05No games: an empty list
The belief covers lists with at least one score, so it predicts nothing.
Reveal the observation
Observed undefined. scores[0] does not exist, and JavaScript gives undefined instead of stopping; the type number does not prevent it.
What this rules in or out Decide whether an empty list is a mistake in the calling code or an input to handle. The assertions section takes this up.
Try the edges first: empty, one, equal, first and last, zero, and negative.
Precisely: noUncheckedIndexedAccess
With the compiler option noUncheckedIndexedAccess, TypeScript gives scores[0] the type number | undefined. Then return low does not compile until the code handles the empty list.
04 / Tests that check nothing
Can a test pass without checking anything?
A test that cannot fail still prints pass. It tells you nothing. Three kinds are common.
The first kind has no comparison. It calls the function and prints the result. Nothing in it can fail.
The second kind gets its expected value from the code under test. It calls the same function again, or it repeats the same formula. So it compares the code with itself.
The third kind has an expected value copied from the output of one run. It checks that the output does not change. If that output was wrong, the test keeps the bug.
The rule from the first section prevents all three: the expected value comes from the belief, worked out without the code. To check a new test, break the code on purpose. For example, make the function return 0 at once, and run the test again. If the test still passes, it does not test that code.
The test below gets want from lowest itself. The experiment runs it on the counterexample from the earlier section.
function testAllWins(): string {
const wins = [2, 1];
const want = lowest(wins);
const got = lowest(wins);
// got === want, whatever lowest returns
...
}A test must be able to fail: its expected value comes from the belief, not from the code.
Precisely: breaking the code on purpose
Mutation testing tools automate the check in this section. They make many small changes to the code, such as < to <=, run the tests after each, and report every change that no test noticed. For TypeScript, Stryker is such a tool.
Experiment
Take the expected value from the function itself
want and got are both results of lowest for the same list. Whatever lowest returns, the two are equal, so the comparison is always true. This test passes with the wrong lowest, and it would pass with a correct one. It cannot tell them apart.
Why it matters Work out the expected value from the belief, by hand. Then a wrong function gives a different value, and the test can fail.
You know that lowest([2, 1]) is wrong. Predict what this test prints. Then replace const want = lowest(wins); with const want = 1; and run again.
function lowest(scores: number[]): number {
let low = 0;
for (const score of scores) {
if (score < low) {
low = score;
}
}
return low;
}
function testAllWins(): string {
const wins = [2, 1];
const want = lowest(wins);
const got = lowest(wins);
let verdict = "pass";
if (got !== want) {
verdict = "FAIL";
}
return verdict + ": got " + got + ", want " + want;
}
console.log(testAllWins());
What to look for
Step into testAllWins. After its two calls to lowest, its frame shows want = 0 and got = 0, and the output is pass: got 0, want 0. After the edit, want is 1, got is still 0, and the output is FAIL: got 0, want 1.
Expected output
pass: got 0, want 005 / Assertions
When should the program stop itself?
A test calls the code from outside, with inputs you chose. An assertion sits inside the code. It states that a condition is true at that point, every time the program reaches it. When the condition is false, the code itself is wrong, and the program should stop with a message.
Use assertions for programmer mistakes. A common one is a broken precondition: something that must be true when a function is called, such as “the list has at least one score”. The code that calls the function is responsible for it. Do not use assertions for problems that come from outside the program, such as a missing file, a score typed as text, or an empty list sent by another program. Those can happen when every line of the code is right. Handle them as errors and tell the user.
JavaScript has no assert statement. console.assert(condition, message) checks the condition. When it is false, console.assert prints Assertion failed: message to standard error. Standard error is a second output stream, for messages about problems; normal output goes to standard output. A terminal shows both, but the stepper shows only standard output. After console.assert, the program continues with the next line.
To stop the program, throw an Error. throw new Error(message) stops the current function and every function that called it. A catch block is code that can stop a thrown error on its way and handle it. When no catch block handles the error, as here, the program ends and prints the message. Below, lowest states its precondition this way.
function lowest(scores: number[]): number {
if (scores.length === 0) {
throw new Error("lowest needs at least one score");
}
let low = scores[0];
// ... the loop as before
return low;
}Throw for what only a programmer mistake can break, because console.assert does not stop the program. Handle everything else as an error.
Coming from Python: Python’s assert stops the program with an AssertionError unless Python runs with -O; console.assert only prints a message, and the program continues.
Coming from Java: JUnit’s assertEquals throws and fails the test; console.assert never throws, so it cannot fail anything.
Precisely: node:assert
Node.js’s assert module, imported from node:assert, has assert(condition, message), which throws an AssertionError when the condition is false. TypeScript also lets a function declare the return type asserts condition, as in function ensure(condition: unknown): asserts condition, so that the type checker knows the condition is true after the call. Both throw. Neither prints a message and continues, as console.assert does.
Experiment
Watch the program continue after console.assert
console.assert reports a false condition and returns, and the next line runs as usual: 0 - 30 is -30. throw stops minutesLeft and the code that called it, so a length of 0 never reaches the subtraction.
Why it matters In JavaScript, an assertion that must stop the program is a throw. console.assert writes a message to standard error, where it is easy to miss.
Game 3’s length is 0, and console.assert checks that a length is above 0. Predict the output. Then replace the console.assert line with if (total <= 0) throw new Error("total must be above 0"); and run again.
function minutesLeft(played: number, total: number): number {
console.assert(total > 0, "total must be above 0");
return total - played;
}
// each game's length in minutes; game 3's was left at 0 by mistake
const totals = [40, 60, 0];
console.log(minutesLeft(30, totals[2]));
What to look for
The stepper stops on the console.assert line with total = 0, then on return total - played, and the output is -30. console.assert did print Assertion failed: total must be above 0, but to standard error, which the stepper does not show. After the edit, the stepper stops on the if line, and the run ends there with Error: total must be above 0, followed by the calls that were running, newest first: at minutesLeft (line 2), then at <top level> (line 8). Nothing is printed. The stepper can stop on the same line more than once; that is normal for this debugger.
Expected output
-3006 / Try it
Can you find the counterexample?
Predict the output, then look for a counterexample before you reveal the answer. A team’s score sheet shows the top scorer of a game, by position in the team list. The belief is the comment at the top of the program.
// topScorer should return the position of the highest score;
// on a tie, the earlier player's position.
function topScorer(points: number[]): number {
let top = 0;
for (let i = 1; i < points.length; i++) {
if (points[i] >= points[top]) {
top = i;
}
}
return top;
}
function check(name: string, got: number, want: number): void {
if (got === want) {
console.log("pass " + name);
} else {
console.log("FAIL " + name + ": got " + got + ", want " + want);
}
}
check("second", topScorer([8, 12, 5]), 1);
check("first", topScorer([20, 3]), 0);
check("alone", topScorer([4]), 0);
What does this print, which input is a counterexample to the belief, and which boundary does it sit on?
Reveal the reasoning
It prints pass second, pass first, and pass alone. The belief is still false: topScorer([9, 9]) returns 1 where the belief gives 0. The counterexample sits on the equal-values boundary: >= moves top to a later player with the same score. A tie below the top, such as [3, 3, 8], would not fail, because 8 wins alone. The repair is > in place of >=, and the tie stays as a regression test.
One more boundary: for an empty team list, the function returns 0, the position of a player who does not exist. The belief needs a decision for that case too.
Practice tickets
- TEST-101 · Long games never finish on the score screen → Free
- TEST-102 · A relay team with one swimmer entered twice was accepted → Sign in · Pro
- TEST-103 · A screen that shows the wrong lift wait passes the checker → Sign in · Pro