Configuration precedence

Before you begin

Why this matters

When a program runs with the wrong setting, the first question is: where did this value come from? If you can follow the order in which your code looks, you can answer that by reading it, and you can spot the lines that replace what someone typed without reporting an error.

What this helps you reason about

Predict which place a setting comes from when several disagree.
Spot code that cannot tell an empty value from a missing one, such as os.Getenv.
Return an error that says which place held the bad value.

A program often needs one setting, such as a port number, that can come from several places. This lesson builds the part of a Go program that picks one value, checks it, and says where it came from.

Already know this? The short version

Take the setting from the highest source that is present, here flag, then environment variable, then file, then the default.
Present means supplied, even as ""; os.Getenv and a one-result map lookup return the zero value "" for a missing name, so use os.LookupEnv or value, ok := m[key].
Check only the chosen text, with your own rule (here 1 to 5 ASCII digits, then 1 to 65535), because strconv.Atoi accepts "+80" and "-80".
An invalid winner is an error that names its source and text; never fall back to a lower source or to the default.
Read flags (flag.Visit tells you which were set), the environment, and files in one place, at the start of the program, and pass their text to a function that reads nothing itself.

Go to the final check· Go to the practice tickets

Words in this lesson (8)

setting: One value a program needs before it can do its work, such as the port number it listens on.
source: A place a setting can come from: a command-line flag, an environment variable, a configuration file, or a default written in the program.
precedence: The fixed order that decides which source wins when several supply the same setting. In this lesson: flag, then environment variable, then file, then default.
present and absent: A source is present when it supplies a value for the setting, even an empty one. It is absent when it supplies nothing at all.
zero value: The value a variable holds when it is declared without one: 0 for numbers, false for bool, "" for string, and nil for pointers, slices, and maps.
comma ok: The two-result form value, ok := m[key]. ok is false when the key is missing, which the value alone cannot show.
validation: Checking that a value follows the rules the program needs, such as “1 to 5 digits, from 1 to 65535”, before using it.
provenance: A record of where a value came from: here, the name of the source that won, kept with the value and named in every error.

Build brief · one port setting

Input: The text of a flag, an environment variable, and a file line, each of which may be absent.
Output: A port number and the source it came from, or an error that names the source and its text.
Policy: flag, then env, then file, then the default 3000. In this program’s policy, a source that is set counts even when its value is empty. Only the winner is checked: 1 to 5 digits, 1 to 65535. An invalid winner is an error.

Examples are focused snippets; imports, enclosing functions, or classes may be omitted. C examples use C11; Haskell examples use Haskell 2010. Expand the notes for language-specific depth.

01 / Precedence

Which source wins?

A setting is one value a program needs before it can do its work, such as the port number it listens on. A setting can come from several sources: a command-line flag such as --port=7000, typed after the program’s name; an environment variable such as APP_PORT, set before the program starts; a line in a configuration file; or a default written in the program.

When several sources supply the port, the program needs a rule. Precedence is that rule: a fixed order of sources in which the first one that supplies a value wins. This program uses a common order: flag, then environment variable, then file, then the default 3000.

For now, the sources are a map from source name to text, and a missing key means an absent source. raw, ok := sources[name] gives two results: the value, and whether the key exists.

main.go

func choose(sources map[string]string) (string, string) {
    for _, name := range []string{"flag", "env", "file"} {
        if raw, ok := sources[name]; ok {
            return name, raw
        }
    }
    return "default", "3000"
}

Look at the sources in precedence order and stop at the first one that supplies a value.

Precisely: the order is a choice

No language or operating system sets this order. Flag, then environment, then file, then default is a common convention, because each source is more specific to one run than the next. Some tools add more sources, such as a file per user and a file per project. Whatever the order, a program should write it down where its users can read it.

Experiment

Follow the sources from the highest

The loop visits the source names in precedence order. raw, ok := sources[name] gives two results: the value, and whether the key exists. choose prints and returns at the first name whose key exists, so the lower names are never looked at.

Why it matters When a setting surprises you, follow the same order by hand. The first source that is present is the one that won.

Predict which source wins before running. Then delete "env": "5000", from the map, so the environment variable is absent too, and predict again.

package main

import "fmt"

func choose(sources map[string]string) {
    for _, name := range []string{"flag", "env", "file"} {
        if raw, ok := sources[name]; ok {
            fmt.Println(raw, "from", name)
            return
        }
    }
    fmt.Println("3000 from default")
}

func main() {
    sources := map[string]string{"env": "5000", "file": "4100"}
    choose(sources)
}

What to look for

Step into choose and watch name. When name is "flag", the next stop is the loop again: the map has no "flag" key. When name is "env", the next stop shows raw "5000" and ok true, and the line after it prints. sources shows its two pairs, "env": "5000" and "file": "4100", in both frames. After the edit, the loop also passes over "env", stops with name "file", and the run prints 4100 from file.

Expected output

5000 from env

02 / Presence

Is an empty value the same as a missing one?

A source is present when it supplies a value, and absent when it supplies nothing. Starting a program with APP_PORT set to "" sets the variable, to an empty value. In this program’s policy, a source that is set counts even when its value is empty: someone set it, so the program checks that empty value and reports an error, instead of moving on to the next source. Some tools choose the other policy and treat an empty variable as not set; either is fine if the documentation says which.

os.Getenv returns the variable’s text, or "", the zero value for a string, when the variable is not set. So its result cannot tell an empty variable from a missing one. os.LookupEnv returns a second result, ok, that is true when the variable is set, even to "". A map lookup with one result, sources["env"], has the same problem as os.Getenv; the two-result form does not.

main.go

// "" when empty and when missing
port := os.Getenv("APP_PORT")
// ok tells the two apart
value, ok := os.LookupEnv("APP_PORT")

Use os.LookupEnv or value, ok := m[key] whenever empty and missing must be told apart.

Coming from Python: os.environ.get("APP_PORT") returns None for a missing variable; os.Getenv returns "", the same as for an empty one.

Experiment

Tell an empty value from a missing one

os.Getenv returns the value of the variable, or "" when the variable is not set, so its result cannot show which happened. os.LookupEnv returns a second result, ok, which is true when the variable is set, even to "".

Why it matters Use os.LookupEnv, or the two-result form of a map lookup, whenever an empty value and a missing one must be told apart.

Predict the three values in the output before running. Then change os.Setenv("APP_PORT", "") to os.Unsetenv("APP_PORT") and predict again.

package main

import (
    "fmt"
    "os"
)

func main() {
    os.Setenv("APP_PORT", "") // as if the shell set APP_PORT to ""
    port := os.Getenv("APP_PORT")
    value, ok := os.LookupEnv("APP_PORT")
    fmt.Printf("[%s] [%s] %t\n", port, value, ok)
}

What to look for

Step past each line. After os.Getenv, port shows "". After os.LookupEnv, value shows "" and ok shows true, and the run prints [] [] true. After the edit, port and value still show "", but ok shows false, and the run prints [] [] false. port is the same in both runs.

Expected output

[] [] true

03 / Validation

What if the winning value is not a port?

The chosen value is still text. Before the program uses it as a port, validation checks the text against the program’s own rule: one to five ASCII digits (the characters 0 to 9), and a number from 1 to 65535. Leading zeros are allowed, so 00080 means 80.

strconv.Atoi is not that rule. It returns an error for "80x" and " 80", but it accepts "+80" and "-80". So the program checks each byte first and converts afterwards. raw[i] is the byte at position i, and comparing it with '0' and '9' tests for an ASCII digit.

When the winning value fails the check, the program could fall through to the next source or stop with an error. This program stops. If the flag says 80x, the person asked for a port on this run. Using the environment’s 5000 instead, with no error, would start the program on a port the person did not ask for.

main.go

digits := len(raw) >= 1 && len(raw) <= 5
for i := 0; i < len(raw); i++ {
    if raw[i] < '0' || raw[i] > '9' {
        digits = false
    }
}
if !digits {
    return 0, source, fmt.Errorf("%s %q is not 1 to 5 digits", source, raw)
}

Check the value that won. If it fails, return an error; do not try the next source.

Coming from JavaScript: Atoi does not return NaN. It returns 0 together with an error, and the 0 looks like a real number unless you check err.

Precisely: ParseUint

strconv.ParseUint(raw, 10, 16) rejects a sign and anything above 65535, but it still accepts 0. Checking the bytes first keeps the rule in one place, and after that check Atoi cannot fail.

Experiment

Give the parser a value that is not a port

Atoi returns an error, together with 0, for text that is not a whole number. Checking err stops that 0 from being used. Atoi also accepts a leading + or -, so "+80" is a number to Atoi, although it is not 1 to 5 digits.

Why it matters Check the text with your own rule before you convert it. When the chosen value fails, report an error; do not swap in another value.

Predict the output before running. Then change "80x" to "+80" and predict again.

package main

import (
    "fmt"
    "strconv"
)

func main() {
    raw, source := "80x", "flag"
    port, err := strconv.Atoi(raw)
    if err != nil {
        fmt.Println(source+":", err)
        return
    }
    fmt.Println(port, "from", source)
}

What to look for

Step past strconv.Atoi. port shows 0, and err shows error(*strconv.NumError), an address, and → {Func: "Atoi", Num: "80x", …} with "invalid syntax" inside. The if takes the error branch, and the run prints flag: strconv.Atoi: parsing "80x": invalid syntax. After the edit, err shows error nil, port shows 80, and the run prints 80 from flag.

Expected output

flag: strconv.Atoi: parsing "80x": invalid syntax

04 / Provenance

How does the program say where a value came from?

Provenance is a record of where a value came from. The program keeps it by returning the source with the port, and by putting the source and the raw text in every error. Then error: flag "80x" is not 1 to 5 digits tells the person which of the four places to fix.

load returns three results: the port, the source, and an error that is nil when everything worked. fmt.Errorf builds an error from a format; %q prints the raw text in quotes, so an empty value shows as "".

main.go

func load(sources map[string]string) (int, string, error)

port, source, err := load(sources)
if err != nil {
    fmt.Println("error:", err)
} else {
    fmt.Println(port, "from", source)
}

Return the source with the value, and name the source and the raw text in every error.

Coming from Java: Go has no exceptions for this: load returns its error as the last result, and the code that called load checks it with if err != nil.

Precisely: showing every setting at start

Many tools can print each setting with the place it came from; git config --list --show-origin is one example. A function that already returns the source makes that a few lines of code.

05 / Real sources

Where do the real flags, variables, and files come in?

A real program reads its sources from outside. The flag package reads the command line, os.LookupEnv reads an environment variable, and os.ReadFile reads a file.

A common Go idiom gives the flag the environment variable as its default: flag.Int("port", envOr("APP_PORT", 3000), "port"), where envOr is a small helper that reads the variable or returns 3000. After flag.Parse, *port holds 5000 whether the person typed -port=5000 or set APP_PORT=5000, so an error can no longer say which one to fix. The second function below keeps the source: it reads the flag as text, and flag.Visit calls its function only for the flags that were given on the command line.

Keep those reads in one place, at the start of the program, and pass the map to load: port, source, err := load(collect()). load itself reads nothing from outside, so you can call it with any combination of sources, as the experiments and the program on this page do.

The stepper on this page always starts a program without flags, and with an environment you cannot set from outside. So this lesson passes the sources in a map. The presence experiment calls os.Setenv inside the program to give APP_PORT a value; a real program would not set its own setting.

main.go

// Not run on this page.

// The common idiom: APP_PORT becomes the flag's default.
// After Parse, *port is 5000 for -port=5000 and for
// APP_PORT=5000 alike, so nothing records which one it was.
func lossy() int {
    port := flag.Int("port", envOr("APP_PORT", 3000), "port")
    flag.Parse()
    return *port
}

// Keeping the source: flag.Visit calls its function only
// for the flags that were set on the command line.
func collect() map[string]string {
    text := flag.String("port", "", "port to listen on")
    flag.Parse()
    sources := map[string]string{}
    flag.Visit(func(f *flag.Flag) {
        if f.Name == "port" {
            sources["flag"] = *text
        }
    })
    if value, ok := os.LookupEnv("APP_PORT"); ok {
        sources["env"] = value
    }
    return sources
}

Read the command line, the environment, and files in one place, and pass plain values to the code that decides.

Precisely: the flag package

flag accepts -port=7000, --port=7000, and -port 7000 with a space, and prints help for -h. flag.Int would reject 80x itself, with its own message, before load could name the source; when the flag is read as a String, load does the check and writes the message. The file is left out here; reading one is a separate topic in this module, Files, parsing, and serialization.

06 / Try it

Can you predict the winner and the outcome?

The workbench below is a model of this lesson’s policy that runs in your browser. It does not run Go code or read your environment. It draws the sources in the reverse of this lesson’s order: flag, the highest source, is at the right end. So the winner is the source furthest to the right that is present.

Predict the result before each change: turn off flag; turn off env as well; turn flag back on and clear its text; then type broken into file while flag holds 7000.

The flag is absent, APP_PORT is set to 0080, and the file says 70000. What does the program report, and why?

Reveal the reasoning

80 from env. The environment variable is the highest source that is present. 0080 is four digits, and leading zeros are allowed, so it gives 80. The file’s 70000 would fail the range check, but the file did not win, so it is never checked.

07 / Whole program

Does the whole program follow the policy?

The complete program below joins the pieces and runs the thirteen acceptance cases, one output line each. Each case is a map with a key for each present source. Copy it into a file and run it with go run to see the same thirteen lines.

main.go

package main

import (
    "fmt"
    "strconv"
)

func load(sources map[string]string) (int, string, error) {
    source, raw := "default", "3000"
    for _, name := range []string{"flag", "env", "file"} {
        if value, ok := sources[name]; ok {
            source, raw = name, value
            break
        }
    }
    digits := len(raw) >= 1 && len(raw) <= 5
    for i := 0; i < len(raw); i++ {
        if raw[i] < '0' || raw[i] > '9' {
            digits = false
        }
    }
    if !digits {
        return 0, source, fmt.Errorf("%s %q is not 1 to 5 digits", source, raw)
    }
    port, _ := strconv.Atoi(raw) // cannot fail: 1 to 5 digits
    if port < 1 || port > 65535 {
        return 0, source, fmt.Errorf("%s %q is not between 1 and 65535", source, raw)
    }
    return port, source, nil
}

func main() {
    // A missing key means that source is absent.
    cases := []map[string]string{
        {"flag": "7000", "env": "5000", "file": "4100"},
        {"env": "5000", "file": "4100"},
        {"file": "4100"},
        {},
        {"flag": "", "env": "5000"},
        {"flag": "80x", "env": "5000"},
        {"flag": " 80"},
        {"flag": "+80"},
        {"flag": "0"},
        {"flag": "65536"},
        {"flag": "7000", "file": "broken"},
        {"flag": "00080"},
        {"env": "", "file": "4100"},
    }
    for _, sources := range cases {
        port, source, err := load(sources)
        if err != nil {
            fmt.Println("error:", err)
        } else {
            fmt.Println(port, "from", source)
        }
    }
}

Acceptance cases

flag 7000, env 5000, file 41007000 from flag
no flag; env 5000, file 41005000 from env
only file 41004100 from file
no flag, env, or file3000 from default
flag "" (set, but empty), env 5000error: flag "" is not 1 to 5 digits
flag 80x, env 5000error: flag "80x" is not 1 to 5 digits
flag " 80" (a space first)error: flag " 80" is not 1 to 5 digits
flag +80error: flag "+80" is not 1 to 5 digits
flag 0error: flag "0" is not between 1 and 65535
flag 65536error: flag "65536" is not between 1 and 65535
flag 7000, file broken7000 from flag; the file is never checked
flag 0008080 from flag
env "" (set, but empty), file 4100error: env "" is not 1 to 5 digits

What does this program settle, and what does it leave out?

Reveal the reasoning

It settles one setting: which source wins, whether its text is a port, and how to say where it came from. It does not read a real command line, environment, or file, check settings that did not win, or start anything on the port. Each of those is a separate piece with its own rules.

Configuration precedence

Why this matters

What this helps you reason about

Which source wins?

Follow the sources from the highest

Is an empty value the same as a missing one?

Tell an empty value from a missing one

What if the winning value is not a port?

Give the parser a value that is not a port

How does the program say where a value came from?

Where do the real flags, variables, and files come in?

Can you predict the winner and the outcome?

Does the whole program follow the policy?

Acceptance cases

Practice tickets

Keep exploring

Primary references