Skip to content

Conformance

The conformance suite is Punix's executable spec. Every claim made elsewhere on this site ("rollback is constant-time", "permuted source yields byte-identical builds", "secret values never reach the store") has a dedicated test in tests/c_e2e/test_conformance_stage*.py that runs end-to-end (no mocks above the eval/realise seam, no subprocess mocks below) and is a CI release-blocker.

A regression in any of these properties blocks the release.

The twelve properties

Property What it asserts Stage Test file
Empty program punix check "" is ok; nothing fetched or built. 0 (implicit)
End-to-end build A real autotools package builds via RealiseLocal, content-addressed, with built + derivation.json; rebuild is a store hit. 3 test_conformance_stage3.py
Order-independence A program and a fully-permuted variant ⇒ byte-identical evaluated values and store paths; only provenance line numbers differ. 2 test_conformance_stage2.py
CVE-pin propagation + isolation force on a shared transitive dep changes exactly its dependents' store paths; unrelated packages are byte-identical. 2 test_conformance_stage2.py
Atomic updates Five sub-properties; see below. 4, 5 test_conformance_stage4.py, test_conformance_stage5_ssh.py
Secret hash-exclusion Two deploys differing only in from_env values ⇒ identical store path; unset secret ⇒ [E11] naming it; no secret value in derivation.json / provenance / store / manifest. 2, 6 test_conformance_stage2.py, test_conformance_stage6.py
Multi-version Ncurses5 / Ncurses6 coexist; a dependent pins one; distinct store paths. 3 test_conformance_stage3.py
Cache correctness Changing a recipe hook body changes the store path. 3 test_conformance_stage3.py
Type-system surface Each [E1]–[E6] has a minimal program producing exactly that code, located. 1 test_conformance_stage1.py
Multi-arch Same inputs on x86_64-linux vs aarch64-linux ⇒ different store paths; a type=url source missing an arch ⇒ [E14]. 6 test_conformance_stage6_multi_arch.py
Migration Migrator on real .rb brew formulae produces type-checking PCL; every shape pattern (cargo / go / npm / autotools / cmake / meson / swift / pip / git-tag source) has a dedicated test. 7 test_migrate_brew.py (95 tests)
Demo deploy The monitoring stack deploys on systemd + docker-compose and (via SSH) to a remote; Grafana on :3000 returns 200. 8 (release evidence) TBD

The atomic-updates sub-properties

The atomic-updates property has five sub-clauses, each with its own dedicated test:

Sub-property What it asserts
Kill mid-deploy The previous generation stays wholly live; current is unchanged.
Rollback exact Rollback after a failed deploy restores the previous gen exactly; the forward gen is preserved on disk.
GC respects pins A GC pass never collects a path pinned by any generation.
Rollback after GC A normal GC pass leaves rollback's target intact.
Pinned-path missing A path that should be there but isn't surfaces a clear [E12] instead of leaving current pointing at gone bytes.

The Stage 5 suite re-runs the first two clauses against SshTransport (different atomic primitive; mv over ssh vs os.replace locally) to prove they're transport-agnostic.

How to run

# Just the conformance suite
uv run pytest tests/c_e2e/test_conformance_stage*.py -v

# A single property
uv run pytest tests/c_e2e/test_conformance_stage4.py::test_20_5_1_kill_mid_deploy_leaves_previous_generation_wholly_live -v

# Everything (the green gate)
make test

Coverage

Stage Tests Properties covered
1 13 type-system surface ([E1]–[E6])
2 4 order-independence, CVE-pin propagation, hash-exclusion at the evaluator
3 6 end-to-end build, multi-version, cache correctness
4 6 atomic-updates (all five clauses) + manifest-schema round-trip
5 3 atomic-updates over SSH + manifest round-trip over SSH
6 9 hash-exclusion at deploy (3) + fixed-output [E13] (3) + multi-arch [E14] (3)

41 conformance tests across the eight stage files, plus 95 migration tests under tests/a_unit/migrate/test_migrate_brew.py driving the brew→PCL translator against every install-block shape. Each is a property statement; the test body is the proof.

The conformance promise

If a conformance test regresses:

  1. The PR is rejected. No "we'll fix this in a follow-up."
  2. The fix lands in the same PR as whatever broke the property.
  3. If the requirement is incorrect, the test update lands in the same PR; the PR description calls out the change in the property.

The conformance suite is the mechanism that makes this policy real. Without it, the policy is just words.