12. Validation Mechanisms

(Normative)

12.1 The Holdout Principle

Governance boundaries — including IP classification boundaries, process compliance boundaries, and delegation authority boundaries — require validation that the implementing agent cannot game. The holdout principle (adapted from machine learning’s holdout set methodology) maintains acceptance criteria that the implementing agent never sees during execution.

The principle: if the implementing agent can see the acceptance criteria, it can optimize for passing them rather than genuinely respecting the boundary. Keeping validation criteria external ensures the implementation serves the governance intent, not the evaluation’s specifics.

12.2 Auto-Research for Boundary Validation

The auto-research pattern (Karpathy, 2026) provides a mechanism for iteratively optimizing governance boundary compliance. The pattern maps to boundary validation as follows:

Auto-Research Element	Boundary Validation Mapping
program.md (human-authored research direction)	Governance boundary specification (the IP classification document, the process compliance spec, etc.)
train.py (agent’s editable workspace)	The boundary enforcement skill or configuration — the thing the agent iteratively improves
prepare.py (frozen evaluation)	The holdout validation scenarios — frozen, cannot be modified by the implementing agent
val_bpb (optimization metric)	Boundary compliance score — binary per scenario, aggregate across scenario set
Experiment loop	Generate test content, run against holdout scenarios, score, refine boundary skill, repeat

The preconditions for auto-research apply: the metric SHALL be legible (boundary compliance is binary and aggregate), the action space SHALL be constrained (the agent edits the boundary skill, not the evaluation), and feedback SHALL be fast (generate and test cycles are cheap).

12.3 Structural Test Categories

The BPM/Agent Stack inherits structural test categories from the Intent Stack’s machine-detectable violations (Annex C.6 of the companion specification):

• Boundary propagation test: The Boundary set at each subprocess SHALL be a superset of the parent’s Boundary set.
• Scope containment test: The authorized activities in each subprocess SHALL be contained within the parent’s authorized scope.
• Policy linkage test: All activities governed by a policy document SHALL include that policy in their Policy Links attribute.
• Vocabulary compliance test: All agent-generated output SHOULD use only terms from the controlled vocabulary.

These tests operate on structure, not content. They MAY be automated at every governance interface and process boundary.