Catnip10

Comparative-genomics pipelines routinely place branch-associated values — rates, residuals, path lengths — into a matrix whose columns are branches of a reference tree. That only means something if a column label keeps denoting the same evolutionary object across every gene that contributes to it.

Genes differ in taxon coverage, so each gene tree is a contraction of the reference tree. Removing a tip can leave an internal node with only two neighbours; standard tree reduction then suppresses that node and merges its two incident edges into one. The merged edge can carry the exact sum of its members' lengths — the arithmetic is perfect — but the two original branches are no longer separately observable in that gene.

If a pipeline writes that merged value into a cell named after one of the two members, the number is numerically valid and the coordinate is not. Nothing crashes, no NA appears, and the error is invisible to every check that only asks whether the arithmetic is correct.

The oracle is deliberately not a phylogenetic method. It is a small graph algorithm that computes the answer by construction, so the benchmark's truth object is exact and inspectable rather than estimated. It never looks at the audited workflow's output.

Primitive identity and branch length are propagated separately: membership by set union and length by addition. Primitive-coordinate state is derived only from the final membership sets.

Historical provenance. Catnip10 grew out of an independent validation layer originally developed for the SplitAligner benchmark. That benchmark directory preserves the historical predecessor. The authoritative Catnip10 oracle specification, truth tables and composite ledgers are maintained in the current Catnip10 repository.

The whole audit turns on separating two kinds of missingness that pipelines habitually collapse into one. A branch that vanished and a branch that merged are not the same event, and they call for different handling downstream.

The first validity axis asks a single question: after pruning and contraction, is a value independently eligible on the primitive full-tree coordinate it is filed under?

The pruning experiment is built so that pruning is the only thing that can explain a discrepancy within this experiment. Topology and branch lengths are fixed. There is no true rate shift, no gene-tree/species-tree discordance, no branch-length estimation error, and no species-tree uncertainty. Taxon deletion and degree-2-node suppression are the only perturbations applied here — the benchmark as a whole also contains representation controls, which vary something else entirely and are described in Axis B below.

Two prespecified deletion schedules give contrasting geometries: a progressive global sequence working inward from the display-root side, and a localized sequence concentrating deletions within one clade. Each runs eight steps including its baseline. Because the tree is small, the oracle can also enumerate the expected state of every coordinate under all subsets of the deletable tips — which is what the interactive panel below is driving.

The second validity axis varies nothing biological at all. It asks whether the same canonical unrooted split keeps the same identity and the same declared availability when the same tree is written down a different way.

Display root. The root location used to serialize or traverse a tree representation. It is not part of the authoritative unrooted branch identity, which remains the canonical split key.

The controls are built as strict equivalents: identical tip sets, identical canonical split keys, and identical branch lengths after canonical alignment. What differs is only how the tree is written.

Auditing a workflow is a mechanical procedure. You normalize one output layer to the adapter contract, declare that layer's exposure, and let the scorer compare it against the oracle ledgers cell by cell. The outcome is one of exactly four dispositions, and only one of them is a failure.

Two branch-wise workflows were run on identical pruned inputs from the global schedule, and their outputs mapped onto the frozen coordinate axis through an explicitly declared exposure. Participant identities are withheld in the public benchmark release; exact identities, versions, adapters and execution provenance are retained in a sealed archive. The two are shown here as Method A and Method B.

One result frames everything below: the native path arithmetic was exact in both workflows. Nothing here is a claim that either computes distances incorrectly. What the audit examines is where those correct numbers are filed.

This case study illustrates the pruning-eligibility axis. Representation stability is tested separately, using the strict-equivalence controls described in Axis B.

Seven progressive deletions from the display-root side. Each tree shows what the contraction does to every coordinate: red dashed where the coordinate becomes structurally absent, blue where it survives only inside a composite, black where it is unaffected.

The independent graph oracle supplies the expected state of every primitive coordinate at every step. This is the truth object the participants are scored against.

Method A declares a fixed shared primitive layer, so the eligibility contract applies to it. Highlighted cells are coordinate-state mismatches in this frozen benchmark configuration: either a coordinate the oracle records as independently observable came back as NA, or a complete composite-path quantity occupies a fusion-ineligible primitive cell.

NOT-EXPOSED to the primitive-axis contract shown here. Method B's native object is endpoint- and path-associated; it is not equivalent to a primitive branch-length matrix and never claims one. The marked cells are not errors. They show how a legitimate native object looks when it is projected onto a frozen primitive axis: typically half of a composite under an endpoint split, or a whole composite attributed to a single endpoint.

The global schedule deletes inward from the display-root side. The localized schedule removes the same number of taxa but concentrates them inside one clade, so fusion and structural absence fall in entirely different places. Everything else is identical: the same reference tree, the same frozen axis, the same declared exposure, the same audit.

The same graph oracle, run on the localized taxon subsets. Note how different this ledger looks from the global one: the coordinates that fuse and the coordinates that vanish are almost a different set.

The same exposed workflow, the same declared fixed-axis contract, and this time no cell disagrees with the oracle. Every coordinate the oracle records as independently observable carries its value; every ineligible cell is empty. Across all seven steps there is nothing to highlight.

Still NOT-EXPOSED to the primitive-axis contract. The estimand boundary has not moved, but the projection signature has: a different geometry produces a different set of marked cells, including single-endpoint full projections into fusion-ineligible cells.

Delete tips and watch the ledger update. Every state shown is the oracle's exact answer for that taxon subset — select a branch to see which coordinates fuse together with it, and which vanish entirely.

SplitAligner constructs and interprets the branch-coordinate framework: it defines branch identity through projected species-tree splits and names every kind of branch absence. Catnip10 tests whether an implementation preserves a declared coordinate contract. The two answer different questions, and neither substitutes for the other.

Catnip10 was developed within the SplitAligner validation ecosystem, which is where the need for an independent truth object first became concrete. Its oracle and scoring contract are implementation-independent: the truth layer is derived by graph contraction, not by asking SplitAligner what the answer should be. SplitAligner is reference infrastructure for the framework, not the benchmark's oracle and not the identity of any participant.

If you want the coordinate system itself — how projected splits define branch identity, and how absence is decomposed — start with the explainer and the tutorial.

You do not have to make your implementation public to be scored. Normalize one output layer to the long-table adapter contract, declare that layer's exposure, and run the scorer. The public repository ships the normative specifications, the frozen benchmark inputs, the oracle truth tables and composite ledgers, machine-readable table schemas, a method-neutral scoring engine, a synthetic example adapter, and anonymous normalized participant outputs. Participant-specific adapters, source versions and execution commands are not public, because they would reveal participant identities.

Axis A requires the complete expected canonical-key set before scoring: missing, duplicate, unknown or inconsistent crosswalk rows are adapter-validation errors rather than participant results. Axis B requires an independently computed certification record before any participant is scored, then aligns outputs by canonical split identity. Synthetic PASS, FAIL, NOT-EXPOSED, true-shift and serialization-only no-op cases are all exercised by the test suite. The authoritative numerical assets are TSV files; rendered documents and figures are views, not the source of truth.

Citing this work. SplitAligner and Catnip10 both rest on two papers and should be cited through both: the SplitAligner main text for the coordinate-construction framework, and the graph-theoretic truth table for the branch-identity definitions and for the uniqueness of the coordinate-and-state ledger the oracle computes. Both are currently available as preprints. The Catnip10 repository may be referenced in addition when the benchmark materials themselves are used. Released under the MIT licence.