Minimal reproductions

Before you begin

Why this matters

When a program gives a wrong answer, the lines that cause it are usually a few among many. If you can shrink the program until only those lines are left, you can often see the cause yourself, and anyone you ask for help can run it in a minute.

What this helps you reason about

Write a failure down precisely enough that someone else can make it happen.
Shrink a failing program one deletion at a time without losing the failure.
Notice when a change made the failure disappear for a reason that has nothing to do with the cause.

Case file: a gym checks whether a locker is booked. The code it builds prints as B7, the booked code is B7, and the check still says no. This lesson shrinks the program until the cause is the only thing left.

Already know this? The short version

Write the expected result, the observed result, and the exact input before guessing; run the failure yourself.
List explanations, each with an observation that would rule it out, and run the cheapest test first.
Delete one piece of code or input at a time and keep a deletion only if the same wrong result remains; replacing a computed string with a literal, or adding final, can make the failure disappear.
Here the cause is == on Strings: it compares references, and a String built while the program runs is a new object, so code == "B7" is false while code.equals("B7") is true.
Write it up with expected and observed results and the Java version, make it print pass or FAIL, and keep it as a check after the fix.

Go to the final check· Go to the practice tickets

Words in this lesson (6)

reproduction: A program and an input that make a failure happen every time they are run.
hypothesis: A possible explanation for a failure, stated so that some observation could show it is wrong.
minimal reproduction: A reproduction from which no single piece of code or input can be deleted without the failure disappearing.
reference: A value that leads to an object, or null. A variable of a class or array type holds a reference, never the object itself, and copying the reference does not copy the object.
== and equals: For two references, == is true when both lead to the same object. equals is a method. A class that overrides it, such as String, compares contents; Object’s own equals compares references, like ==. Two Strings are equal when they contain the same characters.
regression test: A check, kept with the code, that fails while a bug is present and passes once it is fixed, so the bug cannot come back unnoticed.

Case file · B7 is not B7

Expected: Locker B7 is reported as booked.
Observed: The check prints false.
Input: Row "B", number 7; booked code "B7".

Examples are focused snippets; imports, enclosing functions, or classes may be omitted. C examples use C11; Haskell examples use Haskell 2010. Expand the notes for language-specific depth.

01 / The failing case

What exactly is failing?

A gym program builds a locker code from a row letter and a number, and checks it against the booked code. Locker B7 is booked, and the check prints false. Before guessing at a cause, write down three things: what you expected, what you saw, and the exact input. Together they are the failing case. Then run it yourself before changing anything. A failure you have seen happen is evidence. A failure someone only described to you may depend on something they did not mention, so it is not yet evidence.

Here the failing case is: expected true, observed false, input row "B", number 7, and the booked code "B7". In the program, row + number joins the text and the number into one String.

Main.java

static String lockerCode(String row, int number) {
    return row + number;
}

String booked = "B7";
String code = lockerCode("B", 7);
// prints B7
System.out.println(code);
// prints false, expected true
System.out.println(code == booked);

Write the expected result, the observed result, and the exact input before you look for a cause.

Experiment

Run the failing case, then replace the call

lockerCode joins "B" and 7 while the program runs, which makes a new String object. booked is the literal "B7". == asks whether code and booked are the same object, and they are not.

Why it matters A deletion after which the same wrong result remains shows the deleted piece is not needed for the failure. A change that makes the failure disappear shows something about what triggers it. Both are evidence, as long as you change one thing at a time.

Predict both lines. Then replace lockerCode("B", 7) with the literal "B7" and run again.

class Main {
    static String lockerCode(String row, int number) {
        return row + number;
    }

    public static void main(String[] args) {
        String booked = "B7";
        String code = lockerCode("B", 7);
        System.out.println(code);
        System.out.println(code == booked);
    }
}

What to look for

The variables panel shows code = "B7" and booked = "B7", the same text, and the output is B7, then false. The panel shows each string’s text, not which object it is, so the panel alone cannot show the difference. After the edit, the output is B7, then true: the failure disappears although the text is the same, so the difference is in how code was made.

Expected output

B7
false

02 / Explanations

Which explanations fit what you saw?

A hypothesis is a possible explanation, stated so that an observation could show it is wrong. List several before testing any: the first one you think of is often not the cause. For each, write what you would see if it were true, then run the cheapest test first. A test may change the program for a moment; undo the change before the next test. Here are three explanations for the false.

Test the competing explanations

Predict first. These are recorded outcomes of the example, not tests running in your browser.

01Print code

If the code is built wrongly, it does not print B7.

Reveal the observation

Observed It prints B7.

What this rules in or out The text is right.
02Print code.length()

If there is a hidden character, such as a space, the length is not 2.

Reveal the observation

Observed It prints 2.

What this rules in or out There are no hidden characters.
03Print code.equals(booked)

If == compares something other than the text, equals, which compares text, prints true.

Reveal the observation

Observed It prints true.

What this rules in or out == and equals disagree, so == is not comparing the text.

An explanation is worth testing when some observation could rule it out.

03 / Reduce

What can you delete and still see the failure?

Reduce in small steps. Delete one piece: a function, a line, a field, an element of the input. Run again and compare the result with the failing case. If the same wrong result appears, keep the deletion. If the result changes, undo the deletion: that piece is part of the cause, and you have learned something about it.

Only delete while you reduce. Rewriting a line to make it simpler can change what the code does, and then a failure that disappears tells you nothing. Treat the input the same way: shorten it, and check every time that it still fails. You are done when no single deletion keeps the failure. That is a minimal reproduction.

Here lockerCode and booked go: the comparison can use the literal "B7" directly, and building the code in main keeps the failure. Notice what is not a deletion: adding final, or writing "B" + 7 in place of row + 7, changes how the compiler treats the expression, and the experiment shows what that does.

Main.java

String row = "B";
String code = row + 7;
System.out.println(code == "B7"); // false, expected true

Keep a deletion only if the same wrong result is still there.

Precisely: minimal is not smallest

Minimal here means that no single deletion keeps the failure. A different order of deletions can end at a different minimal program, and a cleverer rewrite might be shorter still. Minimal is enough: every line left is needed for the failure, so every line is a clue. Automatic tools that shrink inputs this way exist for many languages; the general method is called delta debugging.

Experiment

Add final and lose the failure

With final, row + 7 is a constant expression: the compiler joins "B" and 7 while compiling and uses the same stored "B7" object as the literal. Without final, the program builds a new String object while it runs.

Why it matters An edit that should change nothing can still change the result. When the failure disappears after a small edit, undo the edit and check that the failure comes back before you draw a conclusion.

Predict the output. Then add final in front of String row, a change that should not alter anything, and run again.

class Main {
    public static void main(String[] args) {
        String row = "B";
        String code = row + 7;
        System.out.println(code == "B7");
    }
}

What to look for

The panel shows row = "B" and code = "B7" in both runs. The output is false. After the edit, the output is true: the failure is gone, and the panel shows no difference.

Expected output

false

04 / The rule

Which rule makes the smallest program fail?

Once the program is small, step through it and find the first line where a value differs from what you expected. Then look up the rule for that line in the documentation.

Here the first surprising value is code == "B7". A String variable holds a reference: a value that leads to an object. == on two references is true when both lead to the same object. equals compares contents: for two Strings, the characters. String literals with the same text share one object, so "B7" == "B7" is true. row + 7 builds a new String object while the program runs, so code and "B7" are two objects with the same text, and == is false.

The bug is not in ==: it does what the Java Language Specification says. The bug was the assumption that == compares text. Many reductions end like this, at a language rule that behaves as documented.

Main.java

String row = "B";
String code = row + 7;
System.out.println(code.equals("B7")); // true

The cause is the first line where a value differs from what you expected; the rule for that line explains why.

Coming from Python: In Python, == on two strings compares their text; in Java, == compares references, and equals compares text.

Coming from C#: C#’s == on two strings compares their text; Java’s == does not, even though the code looks the same.

Precisely: constant expressions

The Java Language Specification (15.29) calls an expression made only of literals and final variables initialized with constant expressions a constant expression. The compiler computes it, and a String constant is interned: every occurrence of the same text shares one object. That is why "B" + 7 and final row + 7 compare as == to "B7". String.intern() does the same while the program runs. Do not rely on any of it for comparisons; use equals.

Experiment

Compare the objects, then the text

== on two String variables compares references: it is true when both lead to one object. equals compares the characters. row + 7 makes a new object, so the two are different objects with equal text until booked is set to code itself.

Why it matters When two values print the same but compare as different, ask what the comparison compares. In Java, use equals for text.

Predict same and equal. Then change String booked = "B7"; to String booked = code; and predict again.

class Main {
    public static void main(String[] args) {
        String row = "B";
        String code = row + 7;
        String booked = "B7";
        boolean same = code == booked;
        boolean equal = code.equals(booked);
        System.out.println(same + " " + equal);
    }
}

What to look for

same shows false and equal shows true; the output is false true. After the edit, booked refers to the same object as code, same becomes true, and the output is true true. In both runs the panel shows the same text for code and booked: same is the visible sign of “the same object”.

Expected output

false true

05 / Write it up

Can someone else run your reproduction?

A reproduction is finished when someone else can run it and see the failure without asking you anything. Write four things: what you expected, what happened, the program, and where you ran it (the language version and the system). Make the program print its own verdict, pass or FAIL, so a reader does not need to know the right answer to read the output.

After the fix, keep the same program as a regression test. A regression is a bug that comes back after it was fixed, and a regression test is a check that fails while the bug is present and passes once it is fixed. If someone later brings the bug back, the check prints FAIL again.

In the program below, condition ? a : b gives a when the condition is true and b otherwise.

Main.java

// Expected true, observed false. OpenJDK 17.
class Main {
    public static void main(String[] args) {
        String row = "B";
        String code = row + 7;
        System.out.println(code == "B7" ? "pass" : "FAIL: == is false");
    }
}

A reproduction prints its own verdict, so anyone can run it and tell whether the bug is still there.

Why is this a better reproduction to send than the program you started with?

Reveal the reasoning

It has nothing left to delete: no lockerCode, no booked variable, and one comparison. A reader sees at once that == on two equal texts is the question. It keeps row as a variable on purpose: replacing it with "B" would hide the failure. It also prints FAIL without the reader knowing the right answer, and after the fix, with equals, it prints pass.

06 / Try it

Can you shrink a fresh failure?

Predict the output, then plan the reduction before you reveal the answer. An event app should find a guest’s position in the guest list.

Main.java

class Example {
    public static void main(String[] args) {
        String[] guests = {"Mia", "Noor"};
        String first = "Mi";
        String wanted = first + "a";
        int found = -1;
        for (int i = 0; i < guests.length; i++) {
            if (guests[i] == wanted) {
                found = i;
            }
        }
        System.out.println("found at " + found);
    }
}

What does this print, which lines can you delete and still see the failure, and what is the smallest input that still fails?

Reveal the reasoning

It prints found at -1, although Mia is at position 0. You can delete Noor, the loop, and found, and print guests[0] == wanted; it still prints false. The smallest reproduction is one comparison between a literal and a String built while the program runs, such as (first + "a") == "Mia". Writing "Mi" + "a" instead would hide the failure, because the compiler joins two literals. The fix is guests[i].equals(wanted).

Minimal reproductions

Why this matters

What this helps you reason about

What exactly is failing?

Run the failing case, then replace the call

Which explanations fit what you saw?

Test the competing explanations

What can you delete and still see the failure?

Add final and lose the failure

Which rule makes the smallest program fail?

Compare the objects, then the text

Can someone else run your reproduction?

Can you shrink a fresh failure?

Practice tickets

Keep exploring

Primary references