Skip to content

Conformance

The conformance suite is Punix's executable spec. Every claim made elsewhere on this site ("rollback is constant-time", "permuted source yields byte-identical builds", "secret values never reach the store") has a dedicated test in tests/c_e2e/test_conformance_stage*.py that runs end-to-end (no mocks above the eval/realise seam, no subprocess mocks below) and is a CI release-blocker.

A regression in any of these properties blocks the release.

The twelve properties

Property What it asserts Stage Test file
Empty program punix check "" is ok; nothing fetched or built. 0 (implicit)
End-to-end build A real autotools package builds via RealiseLocal, content-addressed, with built + derivation.json; rebuild is a store hit. 3 test_conformance_stage3.py
Order-independence A program and a fully-permuted variant ⇒ byte-identical evaluated values and store paths; only provenance line numbers differ. 2 test_conformance_stage2.py
CVE-pin propagation + isolation force on a shared transitive dep changes exactly its dependents' store paths; unrelated packages are byte-identical. 2 test_conformance_stage2.py
Atomic updates Five sub-properties — see below. 4, 5 test_conformance_stage4.py, test_conformance_stage5_ssh.py
Secret hash-exclusion Two deploys differing only in from_env values ⇒ identical store path; unset secret ⇒ [E11] naming it; no secret value in derivation.json / provenance / store / manifest. 2, 6 test_conformance_stage2.py, test_conformance_stage6.py
Multi-version Ncurses5 / Ncurses6 coexist; a dependent pins one; distinct store paths. 3 test_conformance_stage3.py
Cache correctness Changing a recipe hook body changes the store path. 3 test_conformance_stage3.py
Type-system surface Each [E1]–[E6] has a minimal program producing exactly that code, located. 1 test_conformance_stage1.py
Multi-arch Same inputs on x86_64-linux vs aarch64-linux ⇒ different store paths; a type=url source missing an arch ⇒ [E14]. 6 test_conformance_stage6_multi_arch.py
Migration Migrator on real .rb brew formulae produces type-checking PCL; every shape pattern (cargo / go / npm / autotools / cmake / meson / swift / pip / git-tag source) has a dedicated test. 7 test_migrate_brew.py (95 tests)
Demo deploy The monitoring stack deploys on systemd + docker-compose and (via SSH) to a remote; Grafana on :3000 returns 200. 8 (release evidence) TBD

The atomic-updates sub-properties

The atomic-updates property has five sub-clauses, each with its own dedicated test:

Sub-property What it asserts
Kill mid-deploy The previous generation stays wholly live; current is unchanged.
Rollback exact Rollback after a failed deploy restores the previous gen exactly; the forward gen is preserved on disk.
GC respects pins A GC pass never collects a path pinned by any generation.
Rollback after GC A normal GC pass leaves rollback's target intact.
Pinned-path missing A path that should be there but isn't surfaces a clear [E12] instead of leaving current pointing at gone bytes.

The Stage 5 suite re-runs the first two clauses against SshTransport (different atomic primitive — mv over ssh vs os.replace locally) to prove they're transport-agnostic.

How to run

# Just the conformance suite
uv run pytest tests/c_e2e/test_conformance_stage*.py -v

# A single property
uv run pytest tests/c_e2e/test_conformance_stage4.py::test_20_5_1_kill_mid_deploy_leaves_previous_generation_wholly_live -v

# Everything (the green gate)
make test

Coverage

Stage Tests Properties covered
1 13 type-system surface ([E1]–[E6])
2 4 order-independence, CVE-pin propagation, hash-exclusion at the evaluator
3 6 end-to-end build, multi-version, cache correctness
4 6 atomic-updates (all five clauses) + manifest-schema round-trip
5 3 atomic-updates over SSH + manifest round-trip over SSH
6 9 hash-exclusion at deploy (3) + fixed-output [E13] (3) + multi-arch [E14] (3)

41 conformance tests across the eight stage files, plus 95 migration tests under tests/a_unit/migrate/test_migrate_brew.py driving the brew→PCL translator against every install-block shape. Each is a property statement; the test body is the proof.

The conformance promise

If a conformance test regresses:

  1. The PR is rejected. No "we'll fix this in a follow-up."
  2. The fix lands in the same PR as whatever broke the property.
  3. If the property is genuinely wrong (the requirement is incorrect, not the code), the test update lands in the same PR — and the PR description calls out the change in the property.

This is what makes the test suite load-bearing: it is the contract for what Punix does.