Conformance Levels

Purpose #

Explain what Minimum / Standard / Strict mean operationally, why declared-AND-verified matters, and how the harness enforces it.

Big Picture #

Three levels carry over from the upstream orchestrator spec §18 and are codified in .docs/docs/specs/conformance-levels.md. Each adapter and stack pack:

1. Declares a level in its manifest (manifest.conformance).
2. Implements at or above that level.
3. Verifies the claim by running the conformance harness from @aidokit/core in its package CI.

Declared-greater-than-verified is a CI failure. Declared-less-than-verified (over-conforming) is acceptable — the harness reports the actual level.

How It Works #

Level is a capability outcome, not a ranking #

The level an adapter declares is a consequence of what its target CLI can physically support, not a quality grade. Two capabilities drive the ceiling: does the target CLI expose deterministic hooks (for the watchdog), and does it expose output styles? An adapter declares the highest level whose checks its target CLI can actually satisfy.

codex and copilot declare minimum because their target CLIs expose neither hooks nor output styles — not because the adapters are lower quality. claude-code declares strict because Claude Code exposes both. The asymmetry is honest and capability-driven.

The three levels #

A pass at level X means the target passed every check at every level ≤ X.

Where the checks live #

@aidokit/core exports a check table. The canonical id list lives at .docs/docs/specs/conformance-levels.md Appendix B. Each check has a stable id (adapter.emit-verbs.intake-present, stack-pack.skills.has-review-markers, etc.) — these never get renamed once published.

For adapters (≈36 checks total at Strict):

• Manifest checks — declared level, spec version, capabilities, cli-target.
• Agent rules — exactly one file, owner=adapter, source-of-truth chain, validation commands.
• Engine config — paths under engine dir, all relative.
• Verbs — intake / implement-task / orchestrator-next present; intake-stops-before-code.
• Roles — six required (or seven at Standard+); builder is scope-limited-write; read-only roles correct.
• Skills — required skills emitted; triggers preserved; stack-pack skills marked.
• Watchdog — test-loop, command-loop, file-edit counter, repair cap, out-of-scope block, version-bump gate, tracked-deletion gate, blocker emitter, monitoring stream.
• Post-install — executable scripts marked; bd init when Beads enabled.
• Determinism — repeated invocation stable.
• MCP — install / remove / list implemented (MUST at Standard, SHOULD at Minimum).
• Sync / Doctor — pre-sync implemented and complete; doctor implemented and verifies CLI + required files.
• Output styles — present at Strict.

For stack packs (≈21 checks total at Strict):

• Manifest — declared level, languages non-empty, spec version.
• Detect — returns result, confidence categorical, no shell calls, handles missing files.
• Skills — at least one, source marked, review markers, no secrets, no hard-coded paths.
• MCP — references catalog ids, no sensitive suggestions.
• Validation — commands well-formed, ci-safety marked, respects project scripts.
• Architecture — pattern implemented (Strict only).
• Composability — composesWith declared (SHOULD at Strict).
• Determinism.

Where the harness runs #

// packages/adapter-codex/test/conformance.test.ts
import { runAdapterConformance } from '@aidokit/core';
import { adapter } from '../src/index.js';

test(`${adapter.manifest.name} conforms to declared level`, async () => {
  const report = await runAdapterConformance(adapter, adapter.manifest.conformance);
  expect(report.overall).toBe('pass');
});

Mirror pattern for stack packs (runStackPackConformance). Both must be wired into every package's CI; the build fails on any MUST failure.

aidokit doctor integration #

aidokit doctor re-runs a subset of the harness on the user's installed adapter to detect drift between install time and now (<projectRoot>/.aido/conformance-snapshot.json). It does not run the full harness — too slow for user-time.

A MUST check failing during doctor exits with code 13 (CONFORMANCE_FAILED). See .docs/docs/specs/cli-reference.md §10.2.

Example: v1.0 declared levels #

Diagram #

%%{init: {
  "theme": "base",
  "themeVariables": {
    "fontFamily": "ui-sans-serif, system-ui, -apple-system, Segoe UI, sans-serif",
    "fontSize": "14px",
    "primaryColor": "#eff6ff",
    "primaryTextColor": "#0f172a",
    "primaryBorderColor": "#2563eb",
    "lineColor": "#475569",
    "secondaryColor": "#f1f5f9",
    "tertiaryColor": "#ffffff",
    "clusterBkg": "#f8fafc",
    "clusterBorder": "#cbd5e1"
  }
}}%%
flowchart LR
  classDef actor    fill:#ede9fe,stroke:#6d28d9,color:#1e1b4b,stroke-width:1.2px;
  classDef cli      fill:#dbeafe,stroke:#1d4ed8,color:#0c1f4a,stroke-width:1.4px;
  classDef adapter  fill:#cffafe,stroke:#0e7490,color:#083344;
  classDef pack     fill:#dcfce7,stroke:#15803d,color:#052e16;
  classDef core     fill:#fef9c3,stroke:#a16207,color:#422006;
  classDef artifact fill:#f1f5f9,stroke:#475569,color:#0f172a;
  classDef stop     fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d,stroke-dasharray:4 3;
  classDef ok       fill:#ecfdf5,stroke:#047857,color:#064e3b;
  classDef external fill:#fff7ed,stroke:#c2410c,color:#431407;

  declare["manifest.conformance = 'standard'"]:::artifact --> ci{{"runAdapterConformance(adapter, 'standard')"}}:::core
  ci --> pass["overall: pass"]:::ok
  ci --> fail["overall: fail
(any MUST not met)"]:::stop
  fail --> ciFail["CI fails
package cannot publish"]:::stop
  pass --> install["User installs adapter"]:::cli
  install --> snapshot[".aido/conformance-snapshot.json"]:::artifact
  snapshot --> doctor["aidokit doctor (subset run)"]:::cli
  doctor --> drift["drift detected → exit 13"]:::stop
  doctor --> ok["all subset checks pass"]:::ok

See diagrams/system-overview.md for the system-level context.

Common Mistakes #

• Declaring Strict to look good. The harness reads emit method outputs and counts the seventh role / output styles. If they aren't there, CI fails.
• Documenting a gap in prose only. Honest gaps go in manifest.gaps so they surface programmatically. Prose is invisible to tooling.
• Bypassing the harness in CI (e.g. removing the test:conformance script). Adapter publication is prohibited if conformance fails — per .docs/docs/specs/conformance-levels.md §9.3.
• Renaming a check id. Check ids are stable identifiers; once published, they are append-only.

Checklist #

• [ ] Your package's manifest declares a level.
• [ ] Your package has a conformance.test.ts that calls runAdapterConformance / runStackPackConformance.
• [ ] The test runs in CI as part of pnpm test.
• [ ] If your adapter cannot meet a SHOULD/MUST, the gap is in manifest.gaps.

Related Pages #

• ../reference/conformance-reference.md
• ../reference/adapter-contract.md §11
• .docs/docs/specs/conformance-levels.md
• adapters-and-stack-packs.md