Disk-space validation

The disk-space scenario runs each storage node on a checked tmpfs inside an unprivileged user/mount namespace. It fails if namespaces are unavailable; it never substitutes the host disk. Install curl, Python 3, netcat, coreutils, procps and util-linux. Use a machine with at least 4 GiB for licensed EE. The six-node group measured about 1.18 GiB of aggregate node RSS in addition to its 384 MiB tmpfs allocation; allocator, compiler and host services need more.

Build and save separate CE and EE binaries, then run scenarios one at a time:

cargo build --release --bin s4-server
mkdir -p tmp
cp target/release/s4-server tmp/disk-ce-server
cargo build --release --bin s4-server --features enterprise
cp target/release/s4-server tmp/disk-ee-server

KEEP_DEBUG=1 S4_LICENSE_KEY= S4_LICENSE_FILE= \
  bash scripts/63-disk-full-test.sh --groups A,B,C,D,E \
  --root_path tmp/disk-ce-01 ./tmp/disk-ce-server

KEEP_DEBUG=1 S4_LICENSE_KEY= S4_LICENSE_FILE=/path/to/license \
  bash scripts/63-disk-full-test.sh --groups A,B,C,D,E,F \
  --root_path tmp/disk-ee-01 ./tmp/disk-ee-server

Before each run check for leftover scenario processes and use a fresh directory. A/C/E cover safe compaction, hinted delivery and irreversible index poisoning; B races 16 PUTs in each of three rounds; D requires 507 with two full replicas; F requires 200 and RF=3 fallback at EC-set capacity, then controlled tiering of the same object after space returns. Every EE startup verifies the actual edition. The script preserves groups.json, and KEEP_DEBUG=1 retains logs, the JSONL ledger, health, capacity, metrics and verification reports outside the tmpfs. Each group reports preparation, completion, ledger integrity, check identifiers, actual observations, duration and sampled peak node RSS. A cancelled run fails and must release its processes and namespaces.

The ledger has its own fault-detection checks:

python3 -I scripts/tests/disk-full-ledger.py selftest
bash scripts/ci/disk-full-mutations.sh --selftest

Three source mutations verify that the scenario detects the specific defect: client admission disabled (B), coordinator 503 instead of 507 (D), and compaction budget disabled (A). They do not add runtime switches to the server. Run them sequentially with a normal CE baseline, after building the pinned Rust toolchain:

bash scripts/ci/disk-full-mutations.sh --baseline ./tmp/disk-ce-server \
  --root_path tmp/disk-mutations-01 --seed-target target

--seed-target optionally copies an existing release cache; it never links or overwrites normal artifacts. Each mutation has an independent source copy and a saved binary. The report preserves the base SHA, working-source diff, exact patches, build parameters, binary hashes, positive and negative group reports. Only the expected failing check in a completed group with successful setup and an intact ledger counts as detection. Build failures, timeouts, crashes, lost acknowledged data and unrelated failures invalidate the control. --mutant-binary NAME=PATH can evaluate a prepared mutant; supplying the ordinary baseline is explicitly rejected. The historical --unfixed mode is separate: it applies only to A/C/E against a build predating disk admission.

The private workflow runs positive CE/EE groups before mutation controls and keeps evidence for 14 days. The public publishing workflow does not include these tests because it does not publish the private scenario sources.