Tests

Tests live in the file with the code, run with one command, and are checked by the same compiler with the same rules. A test that does not compile is a failing test; a test that leaks is reported by nx leaks; a test that could have run at compile time, can.

topo/code/testing.nx

import std.testing

/// Celsius from Fahrenheit.
fn to_celsius(f: f64) -> f64 { return (f - 32.0) * 5.0 / 9.0 }

/// The median of a sorted slice; null when it is empty.
fn median(sorted: []i32) -> ?i32 {
    if sorted.len == 0 { return null }
    let mid = sorted.len / 2
    if sorted.len % 2 == 1 { return sorted[mid] }
    return (sorted[mid - 1] + sorted[mid]) / 2
}

error Parse { Bad }

fn parse_bool(s: []u8) -> Parse!bool {
    if s == "yes" { return true }
    if s == "no" { return false }
    return error.Bad
}

fn main() {
    var readings: [3]i32 = [3, 1, 2]
    readings[..].sort()
    println("{:.1} {}", .{to_celsius(212.0), median(readings[..]) orelse -1})
}

test "freezing and boiling" {
    expect_eq(to_celsius(32.0), 0.0)
    expect_eq(to_celsius(212.0), 100.0)
}

test "body temperature, approximately" {
    testing.expect_approx(to_celsius(98.6), 37.0, 0.01)
}

test "median of odd, even and empty" {
    let odd = [1, 2, 3]
    let even = [1, 2, 3, 4]
    expect_eq(median(odd[..]) orelse -1, 2)
    expect_eq(median(even[..]) orelse -1, 2)
    let empty = List(i32).new()
    expect(median(empty[..]) == null)
}

test "errors" {
    expect(try parse_bool("yes"))
    testing.expect_error(bool, parse_bool("maybe"), error.Bad)
}

test "text" {
    testing.expect_contains("base camp", "camp")
    testing.expect_lines("a\nb\n", "a\nb\n")
}

/// Up to nine readings between -40 and 40, for the property below.
fn readings(r: *mut testing.Rng) -> List(i32) {
    var out = List(i32).new()
    for _ in 0..r.size(9) { out.append(r.int(-40, 40) as i32) }
    return out
}

test "the median lies between the smallest and the largest" {
    testing.check(List(i32), readings, |xs: *List(i32)| -> bool {
        var sorted = xs.clone()
        sorted[..].sort()
        let m = median(sorted[..]) orelse return true
        return m >= sorted[0] and m <= sorted[sorted.len - 1]
    })
}

comptime test "runs while compiling" {
    expect_eq(median([5][..]) orelse 0, 5)
}

$ nx test topo/code/testing.nx
ok    freezing and boiling
ok    body temperature, approximately
ok    median of odd, even and empty
ok    errors
ok    text
ok    the median lies between the smallest and the largest

6 passed, 0 failed

`test` blocks#

test "freezing and boiling" {
    expect_eq(to_celsius(32.0), 0.0)

A test "name" { } block is a function with no arguments that nx test runs, in file order, each in its own process state. expect(cond) and expect_eq(a, b) are builtins; a failed expectation panics with the values and the location, the test is reported as failed, and the run continues with the next one. nx run ignores test blocks entirely, so they cost the program nothing.

Inside a test, try is allowed (a test that hits an unexpected error fails with the error's name), defer works, and any function in the file is reachable, pub or not. Every module of the standard library is tested this way, in its own file: nx test std/strings.nx runs the strings module's tests.

`std.testing`#

The builtins cover most tests. std.testing adds the assertions that are tedious to write by hand:

expect_approx(a, b, eps) for floats.
expect_err(T, r) and expect_error(T, r, error.Name) for results that should fail.
expect_contains(text, needle) and expect_lines(actual, expected), which names the first line that differs.
expect_snapshot(name, actual): compares actual with snapshots/<name>.txt, creates the file the first time, and rewrites it when NX_UPDATE_SNAPSHOTS=1 is set; a mismatch names the file and the first line that differs. try snapshot(name, actual) is the same as an error union. Snapshot tests are how a program's whole output is kept honest with one line.

Properties#

An example checks one input; a property states a rule and lets the machine look for an input that breaks it. testing.check(T, gen, holds) asks gen for 100 values and fails the test when holds is false of one:

test "the median lies between the smallest and the largest" {
    testing.check(List(i32), readings, |xs: *List(i32)| -> bool {

gen draws from a testing.Rng: r.int(lo, hi), r.size(max) for a length, r.pick(n), r.flip(), r.float(), r.bytes(max), r.ascii(max), and r.text(max) for UTF-8 with accents, CJK and emoji in it. The Rng keeps every choice it made, and that makes a failure worth reading: the failing case is made again with its choices cut short and lowered for as long as it still fails, so the report is about the smallest case found, a list of two zeros rather than of fifteen random numbers, with no shrinking code of your own. check_show writes that case with a function you give it. The failure names its seed, and NX_SEED=<seed> nx test runs the same cases again (NX_CASES says how many). testing.search is the same driver returning what it found instead of failing the test, and the compiler's fuzzer, tests/fuzz.nx, runs on it: what it finds is shrunk before it is saved.

The command#

nx test file.nx                    # every test in the file
nx test file.nx --filter median    # only the tests whose name contains it
nx test file.nx --verbose          # with timings

The exit code is the number of failures, so CI sees a red run without parsing anything.

Compile-time tests#

comptime test "runs while compiling" {
    expect_eq(median([5][..]) orelse 0, 5)
}

The last chapter introduced these. A comptime test runs in the interpreter during checking; it is not listed by nx test because it already ran when the file was checked, and a failure is a compile error. Use them for pure functions whose examples are worth stating next to the code, and for facts about constants.

What the test suite of this book looks like#

The programs in these chapters are tested by the same harness that tests the compiler: nx run tests/run.nx -- topo runs every one, compares its output with the recorded .expected file the page shows, runs the test blocks in the calculator and this chapter's file, builds the packages example from its own directory, renders the GUI chapter's frame, and builds this site. The harness is a Nexium program, tests/run.nx, which is worth a read once you have finished the book: it is a 400-line example of process control, threads and file handling in the language.

Next: effects, the feature that is Nexium's own.

Exercises#

The compiler grades these: nx topo 14 writes them into a directory, nx topo check runs them. Or do them here: the code is editable, Run runs it with the compiler in the page (it loads on the first run and never sends your code anywhere), Check grades it as nx topo check would, and Reset puts the exercise back. Ctrl+Enter runs.

test_clampmake the tests pass

One test is wrong about clamp, and clamp is wrong about one side. Fix both.

test_clamp.nx

fn clamp(x: i32, lo: i32, hi: i32) -> i32 {
    if x < lo { return lo }
    if x > hi { return x }
    return x
}

test "inside" { expect_eq(clamp(5, 0, 10), 5) }
test "below" { expect_eq(clamp(-3, 0, 10), 0) }
test "above" { expect_eq(clamp(42, 0, 10), 10) }
test "at the edges" {
    expect_eq(clamp(0, 0, 10), 0)
    expect_eq(clamp(10, 0, 10), 11)
}

Hints

nx test names the failing expectation and its line

Solution

fn clamp(x: i32, lo: i32, hi: i32) -> i32 {
    if x < lo { return lo }
    if x > hi { return hi }
    return x
}

test "inside" { expect_eq(clamp(5, 0, 10), 5) }
test "below" { expect_eq(clamp(-3, 0, 10), 0) }
test "above" { expect_eq(clamp(42, 0, 10), 10) }
test "at the edges" {
    expect_eq(clamp(0, 0, 10), 0)
    expect_eq(clamp(10, 0, 10), 10)
}

done

write_is_sortedwrite it

Fill in is_sorted so the tests pass: true when every element is at most the next one.

write_is_sorted.nx

fn is_sorted(xs: []i32) -> bool {
    ???
}

test "sorted" {
    let a: [4]i32 = [1, 2, 2, 9]
    expect(is_sorted(a[..]))
}

test "not sorted" {
    let a: [3]i32 = [3, 1, 2]
    expect(!is_sorted(a[..]))
}

test "short" {
    let one: [1]i32 = [7]
    expect(is_sorted(one[..]))
    let none = List(i32).new()
    expect(is_sorted(none[..]))
}