Conformance Levels

Purpose #

Explain what Minimum / Standard / Strict mean operationally, why declared-AND-verified matters, and how the harness enforces it.

Big Picture #

Three levels carry over from the upstream orchestrator spec §18 and are codified in .docs/docs/specs/conformance-levels.md. Each adapter and stack pack:

1. Declares a level in its manifest (manifest.conformance).
2. Implements at or above that level.
3. Verifies the claim by running the conformance harness from @aidokit/core in its package CI.

Declared-greater-than-verified is a CI failure. Declared-less-than-verified (over-conforming) is acceptable — the harness reports the actual level.

How It Works #

The three levels #

A pass at level X means the target passed every check at every level ≤ X.

Where the checks live #

@aidokit/core exports a check table. The canonical id list lives at .docs/docs/specs/conformance-levels.md Appendix B. Each check has a stable id (adapter.emit-verbs.intake-present, stack-pack.skills.has-review-markers, etc.) — these never get renamed once published.

For adapters (≈36 checks total at Strict):

• Manifest checks — declared level, spec version, capabilities, cli-target.
• Agent rules — exactly one file, owner=adapter, source-of-truth chain, validation commands.
• Engine config — paths under engine dir, all relative.
• Verbs — intake / implement-task / orchestrator-next present; intake-stops-before-code.
• Roles — six required (or seven at Standard+); builder is scope-limited-write; read-only roles correct.
• Skills — required skills emitted; triggers preserved; stack-pack skills marked.
• Watchdog — test-loop, command-loop, file-edit counter, repair cap, out-of-scope block, version-bump gate, tracked-deletion gate, blocker emitter, monitoring stream.
• Post-install — executable scripts marked; bd init when Beads enabled.
• Determinism — repeated invocation stable.
• MCP — install / remove / list implemented (MUST at Standard, SHOULD at Minimum).
• Sync / Doctor — pre-sync implemented and complete; doctor implemented and verifies CLI + required files.
• Output styles — present at Strict.

For stack packs (≈21 checks total at Strict):

• Manifest — declared level, languages non-empty, spec version.
• Detect — returns result, confidence categorical, no shell calls, handles missing files.
• Skills — at least one, source marked, review markers, no secrets, no hard-coded paths.
• MCP — references catalog ids, no sensitive suggestions.
• Validation — commands well-formed, ci-safety marked, respects project scripts.
• Architecture — pattern implemented (Strict only).
• Composability — composesWith declared (SHOULD at Strict).
• Determinism.

Where the harness runs #

// packages/adapter-claude-code/test/conformance.test.ts
import { runAdapterConformance } from '@aidokit/core';
import { adapter } from '../src/index.js';

test(`${adapter.manifest.name} conforms to declared level`, async () => {
  const report = await runAdapterConformance(adapter, adapter.manifest.conformance);
  expect(report.overall).toBe('pass');
});

Mirror pattern for stack packs (runStackPackConformance). Both must be wired into every package's CI; the build fails on any MUST failure.

aidokit doctor integration #

aidokit doctor re-runs a subset of the harness on the user's installed adapter to detect drift between install time and now (<projectRoot>/.aido/conformance-snapshot.json). It does not run the full harness — too slow for user-time.

A MUST check failing during doctor exits with code 13 (CONFORMANCE_FAILED). See .docs/docs/specs/cli-reference.md §10.2.

Example: v1.0 declared levels #

Diagram #

%%{init: {
  "theme": "base",
  "themeVariables": {
    "fontFamily": "ui-sans-serif, system-ui, -apple-system, Segoe UI, sans-serif",
    "fontSize": "14px",
    "primaryColor": "#eff6ff",
    "primaryTextColor": "#0f172a",
    "primaryBorderColor": "#2563eb",
    "lineColor": "#475569",
    "secondaryColor": "#f1f5f9",
    "tertiaryColor": "#ffffff",
    "clusterBkg": "#f8fafc",
    "clusterBorder": "#cbd5e1"
  }
}}%%
flowchart LR
  classDef actor    fill:#ede9fe,stroke:#6d28d9,color:#1e1b4b,stroke-width:1.2px;
  classDef cli      fill:#dbeafe,stroke:#1d4ed8,color:#0c1f4a,stroke-width:1.4px;
  classDef adapter  fill:#cffafe,stroke:#0e7490,color:#083344;
  classDef pack     fill:#dcfce7,stroke:#15803d,color:#052e16;
  classDef core     fill:#fef9c3,stroke:#a16207,color:#422006;
  classDef artifact fill:#f1f5f9,stroke:#475569,color:#0f172a;
  classDef stop     fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d,stroke-dasharray:4 3;
  classDef ok       fill:#ecfdf5,stroke:#047857,color:#064e3b;
  classDef external fill:#fff7ed,stroke:#c2410c,color:#431407;

  declare["manifest.conformance = 'standard'"]:::artifact --> ci{{"runAdapterConformance(adapter, 'standard')"}}:::core
  ci --> pass["overall: pass"]:::ok
  ci --> fail["overall: fail
(any MUST not met)"]:::stop
  fail --> ciFail["CI fails
package cannot publish"]:::stop
  pass --> install["User installs adapter"]:::cli
  install --> snapshot[".aido/conformance-snapshot.json"]:::artifact
  snapshot --> doctor["aidokit doctor (subset run)"]:::cli
  doctor --> drift["drift detected → exit 13"]:::stop
  doctor --> ok["all subset checks pass"]:::ok

See diagrams/system-overview.md for the system-level context.

Common Mistakes #

• Declaring Strict to look good. The harness reads emit method outputs and counts the seventh role / output styles. If they aren't there, CI fails.
• Documenting a gap in prose only. Honest gaps go in manifest.gaps so they surface programmatically. Prose is invisible to tooling.
• Bypassing the harness in CI (e.g. removing the test:conformance script). Adapter publication is prohibited if conformance fails — per .docs/docs/specs/conformance-levels.md §9.3.
• Renaming a check id. Check ids are stable identifiers; once published, they are append-only.

Checklist #

• [ ] Your package's manifest declares a level.
• [ ] Your package has a conformance.test.ts that calls runAdapterConformance / runStackPackConformance.
• [ ] The test runs in CI as part of pnpm test.
• [ ] If your adapter cannot meet a SHOULD/MUST, the gap is in manifest.gaps.

Related Pages #

• ../reference/conformance-reference.md
• ../reference/adapter-contract.md §11
• .docs/docs/specs/conformance-levels.md
• adapters-and-stack-packs.md