Before you begin
Why this matters
When a program gives a wrong answer, the lines that cause it are usually a few among many. If you can shrink the program until only those lines are left, you can often see the cause yourself, and anyone you ask for help can run it in a minute.
What this helps you reason about
- Write a failure down precisely enough that someone else can make it happen.
- Shrink a failing program one deletion at a time without losing the failure.
- Notice when a change made the failure disappear for a reason that has nothing to do with the cause.
Case file: a weather program reports the wettest day of the week, and for one week it reports the wrong one. Nothing crashes. This lesson shrinks the program until the cause is the only thing left.
Already know this? The short version
- Write the expected result, the observed result, and the exact input before guessing; run the failure yourself.
- List explanations, each with an observation that would rule it out, and run the cheapest test first.
- Delete one piece of code or input at a time and keep a deletion only if the same wrong result remains; reordering or shortening an input can remove the property that triggers the failure.
- Here the cause is an off-by-one loop bound: i < days - 1 never visits the last index, so {1, 2} gives 1; the bound for every index is i < days.
- Write it up with expected and observed results, the compiler and its options, make it print pass or FAIL, and keep it as a check after the fix.
Words in this lesson (6)
- reproduction
- A program and an input that make a failure happen every time they are run.
- hypothesis
- A possible explanation for a failure, stated so that some observation could show it is wrong.
- minimal reproduction
- A reproduction from which no single piece of code or input can be deleted without the failure disappearing.
- off-by-one error
- A mistake where a loop or a count goes one step too far or stops one step too early, such as a loop that never reaches the last element of an array.
- undefined behavior
- Code for which the C standard sets no requirements. The program may crash, print wrong results, or seem to work, and the result can change between builds.
- regression test
- A check, kept with the code, that fails while a bug is present and passes once it is fixed, so the bug cannot come back unnoticed.
Case file · the wettest day is missing
- Expected
- The wettest day had 15 mm of rain.
- Observed
- The program reports 12 mm.
- Input
- Rain per day: 3, 0, 12, 4, 0, 7, 15.
Examples are focused snippets; imports, enclosing functions, or classes may be omitted. C examples use C11; Haskell examples use Haskell 2010. Expand the notes for language-specific depth.
01 / The failing case
What exactly is failing?
A weather program reports the most rain that fell on one day of the week. For a week ending with 15 mm, it reports 12. Before guessing at a cause, write down three things: what you expected, what you saw, and the exact input. Together they are the failing case. Then run it yourself before changing anything. A failure you have seen happen is evidence. A failure someone only described to you may depend on something they did not mention, so it is not yet evidence.
Here the failing case is: expected 15, observed 12, input the days 3, 0, 12, 4, 0, 7, 15 in millimetres. In the program, mm is the array of daily rain and days is how many days it holds; const means wettest only reads the array.
int wettest(const int mm[], int days) {
int most = mm[0];
for (int i = 1; i < days - 1; i++) {
if (mm[i] > most) {
most = mm[i];
}
}
return most;
}
/* wettest(week, 7) is 12, expected 15 */Write the expected result, the observed result, and the exact input before you look for a cause.
Experiment
Run the failing case, then change one day
The loop starts at the second day and runs while i < days - 1, so it stops before the last day. most starts at the first day, so every day except the last is compared.
Why it matters A deletion after which the same wrong result remains shows the deleted piece is not needed for the failure. A change that makes the failure disappear shows something about what triggers it. Both are evidence, as long as you change one thing at a time.
Predict the wettest day the program reports. Then change the last day’s 15 to 5 and run again. Is the failure still there?
#include <stdio.h>
int wettest(const int mm[], int days) {
int most = mm[0];
for (int i = 1; i < days - 1; i++) {
if (mm[i] > most) {
most = mm[i];
}
}
return most;
}
int main(void) {
int week[] = {3, 0, 12, 4, 0, 7, 15};
printf("wettest day: %d mm\n", wettest(week, 7));
return 0;
}
What to look for
Step into wettest and watch i on the if line: it takes the values 1 to 5 and never 6, so the last day, mm[6], is never compared. most ends at 12. mm shows an address and the value stored there, 3: it points to the first day of week. The output, wettest day: 12 mm, appears when the run ends, because under the stepper C output waits until the program finishes. After the edit, the output is still 12, which is now the right answer: the failure disappears when the wettest day is not the last one.
Expected output
wettest day: 12 mm02 / Explanations
Which explanations fit what you saw?
A hypothesis is a possible explanation, stated so that an observation could show it is wrong. List several before testing any: the first one you think of is often not the cause. For each, write what you would see if it were true, then run the cheapest test first. A test may change the program for a moment; undo the change before the next test. Here are three explanations for the wrong day.
Test the competing explanations
Predict first. These are recorded outcomes of the example, not tests running in your browser.
01Look at week in main
If the array was declared one element short, or 15 was never stored, week does not show it.
Reveal the observation
Observed week shows {3, 0, 12, 4, 0, 7, 15}.
What this rules in or out The input is right.
02Change the first day to 20
If the loop skips the first day, the result is not 20.
Reveal the observation
Observed The result is 20.
What this rules in or out The first day is counted, because most starts at mm[0].
03Watch i in wettest
If the loop misses the last day, i never reaches 6.
Reveal the observation
Observed i takes the values 1 to 5.
What this rules in or out The loop stops one day early.
An explanation is worth testing when some observation could rule it out.
03 / Reduce
What can you delete and still see the failure?
Reduce in small steps. Delete one piece: a function, a line, a field, an element of the input. Run again and compare the result with the failing case. If the same wrong result appears, keep the deletion. If the result changes, undo the deletion: that piece is part of the cause, and you have learned something about it.
Only delete while you reduce. Rewriting a line to make it simpler can change what the code does, and then a failure that disappears tells you nothing. Treat the input the same way: shorten it, and check every time that it still fails. You are done when no single deletion keeps the failure. That is a minimal reproduction.
Here the week shrinks while the wettest day stays last: {1, 2} with two days still gives the wrong answer. One day, {5}, would not fail, because most starts at the first day and the loop has nothing to skip.
int week[] = {1, 2};
printf("%d\n", wettest(week, 2)); /* 1, expected 2 */Keep a deletion only if the same wrong result is still there.
Precisely: minimal is not smallest
Minimal here means that no single deletion keeps the failure. A different order of deletions can end at a different minimal program, and a cleverer rewrite might be shorter still. Minimal is enough: every line left is needed for the failure, so every line is a clue. Automatic tools that shrink inputs this way exist for many languages; the general method is called delta debugging.
Experiment
Reorder the input and lose the failure
The loop runs while i < days - 1. With two days that is i < 1, so the loop body never runs and most stays at the first value. The failure needs the largest value in the last place.
Why it matters Reducing an input can change its order or move its largest value. Check after each change that the property that triggers the failure, here “the largest value is last”, is still there.
Predict what wettest returns for {1, 2}. Then swap the two values to {2, 1} and predict again.
#include <stdio.h>
int wettest(const int mm[], int days) {
int most = mm[0];
for (int i = 1; i < days - 1; i++) {
if (mm[i] > most) {
most = mm[i];
}
}
return most;
}
int main(void) {
int week[] = {1, 2};
printf("%d\n", wettest(week, 2));
return 0;
}
What to look for
After most = mm[0], the stepper never stops inside the loop: it goes from the for line to return most, and the output is 1. After the edit, the output is 2, the right answer. With the larger value first, mm[0] is already the answer, so a loop that skips the last day does no harm.
Expected output
104 / The rule
Which rule makes the smallest program fail?
Once the program is small, step through it and find the first line where a value differs from what you expected. Then look up the rule for that line in the documentation.
Here the value to watch is i. An array of days elements has indexes 0 to days - 1, so a loop that visits every index from 1 runs while i < days. The condition i < days - 1 stops one index early and never visits the last day. This is an off-by-one error: a loop or a count that goes one step too far or stops one step too early.
Nothing in C checks loop bounds. Going one step too far would read past the end of the array, which is undefined behavior: the program may crash, print wrong results, or seem to work. Stopping one step too early, as here, gives a wrong answer every time the last day is the wettest.
for (int i = 1; i < days; i++) {
if (mm[i] > most) {
most = mm[i];
}
}The cause is the first line where a value differs from what you expected; the rule for that line explains why.
Coming from Python: Python’s max(week) cannot skip a day; in C you write the loop bounds yourself, and the compiler does not check them.
Coming from Java: Java throws ArrayIndexOutOfBoundsException when a loop goes one step too far; C does not check, and a loop that stops one step too early is wrong in every language.
Precisely: the other direction
If the bound were i <= days, the loop would read mm[days], one element past the end. That is undefined behavior, so a reproduction of it may print a different wrong value on another build or seem to work. Build such a reproduction with -fsanitize=address (gcc and clang): AddressSanitizer stops the program at the read past the end and reports where it happened.
Experiment
Watch the loop stop one day early
With days = 3, i < days - 1 is i < 2, so the loop body runs for i = 1 only. The last index is days - 1, which is 2, and i < days is the condition that reaches it.
Why it matters A loop over an array of n elements ends with i < n. When a result is wrong only for some inputs, check whether the first or the last element is ever visited.
Predict how many times the if line runs for {4, 1, 9}. Then change i < days - 1 to i < days and predict again.
#include <stdio.h>
int main(void) {
int mm[] = {4, 1, 9};
int days = 3;
int most = mm[0];
for (int i = 1; i < days - 1; i++) {
if (mm[i] > most) {
most = mm[i];
}
}
printf("%d\n", most);
return 0;
}
What to look for
The stepper stops on the if line once, with i = 1, and most stays 4; the output is 4. After the edit it also stops there with i = 2, most changes to 9, and the output is 9.
Expected output
405 / Write it up
Can someone else run your reproduction?
A reproduction is finished when someone else can run it and see the failure without asking you anything. Write four things: what you expected, what happened, the program, and where you ran it (the language version and the system). Make the program print its own verdict, pass or FAIL, so a reader does not need to know the right answer to read the output.
After the fix, keep the same program as a regression test. A regression is a bug that comes back after it was fixed, and a regression test is a check that fails while the bug is present and passes once it is fixed. If someone later brings the bug back, the check prints FAIL again.
/* Expected 2, observed 1. gcc 12 at -O0, Linux. */
#include <stdio.h>
int wettest(const int mm[], int days) {
int most = mm[0];
for (int i = 1; i < days - 1; i++) {
if (mm[i] > most) {
most = mm[i];
}
}
return most;
}
int main(void) {
int week[] = {1, 2};
int got = wettest(week, 2);
if (got == 2) {
printf("pass\n");
} else {
printf("FAIL: got %d\n", got);
}
return 0;
}A reproduction prints its own verdict, so anyone can run it and tell whether the bug is still there.
Why is this a better reproduction to send than the program you started with?
Reveal the reasoning
It has nothing left to delete from the input: two days, with the wetter one last. A reader sees at once that the second day is ignored, and does not have to compare seven numbers. wettest stays whole because it is the code under suspicion. The program also prints FAIL without the reader knowing the right answer, and after the fix it prints pass.
06 / Try it
Can you shrink a fresh failure?
Predict the output, then plan the reduction before you reveal the answer. A weather program should count the days of the week hotter than 30 degrees.
#include <stdio.h>
int main(void) {
int temps[] = {31, 24, 33, 28};
int days = 4;
int hot = 0;
for (int i = 1; i < days; i++) {
if (temps[i] > 30) {
hot++;
}
}
printf("hot days: %d\n", hot);
return 0;
}
What does this print, which lines can you delete and still see the failure, and what is the smallest input that still fails?
Reveal the reasoning
It prints hot days: 1, where 31 and 33 make 2. You can delete the days that are not hot and the day of 33, as long as the hot first day stays; the count stays one short. The smallest input is a single hot day, {31}, which gives 0. A single hot day placed second would not fail, because only the first day is skipped. The loop starts at i = 1; the fix is i = 0.
Practice tickets
- REPRO-101 · Fix the longest streak → Free
- REPRO-102 · Pin the parking charge with a test → Sign in · Pro
- REPRO-103 · Shrink a crashing formula → Sign in · Pro
Keep exploring
Assertions, tests, and counterexamples →