Conformance Levels Specification
Formal specification of the three conformance levels (
Minimum/Standard/Strict), the per-level checklists each level requires, and the verification harness that enforces them.
| Field | Value |
|---|---|
| Spec name | aidokit Conformance Levels |
| Spec version | 1.0 (draft) |
| Status | Pre-alpha — checklists frozen at spec freeze; harness implementation evolves |
| Implements | @aidokit/core conformance harness |
| Sibling specs | adapter-contract.md, stack-pack-contract.md |
| Last reviewed | Pre-publication |
Table of contents
- Purpose and audience
- Conventions
- Conformance model
- Adapter conformance — per-level checklists
- Stack-pack conformance — per-level checklists
- The conformance harness
- Test fixtures
- Reporting format
- CI integration
aidokit doctorintegration- Project-level conformance
- Trust signals and marketplace
- Failure modes and recovery
- Versioning and evolution
- Open questions
- Appendix A: Synthetic ProjectContext fixtures
- Appendix B: Full check id reference
1. Purpose and audience
1.1 Purpose
This specification defines:
- The three conformance levels (
Minimum,Standard,Strict) operationally - The exact checklist a target (adapter or stack pack) must pass at each level
- The harness API that runs these checks
- The reporting format produced by the harness
- How
aidokit doctorre-verifies conformance at user time - How conformance maps to marketplace trust signals
adapter-contract.md §11 and stack-pack-contract.md §15 declare which methods are MUST/SHOULD/MAY at each level. This spec aggregates those declarations into executable checklists that the harness implements. If there is a conflict, the contract specs win for definitions; this spec wins for verification details.
1.2 Audience
- Adapter and stack-pack authors running the harness in their package CI
@aidokit/coredevelopers implementing the harness@aidokit/clidevelopers invoking the harness viaaidokit doctor- Reviewers auditing a conformance claim
- Marketplace tooling generating trust badges from harness outputs
1.3 Non-purpose
This document does NOT:
- Re-define the Adapter or StackPack interfaces (see contract specs)
- Define the trigger grammar (see
mcp-catalog.md) - Define the CLI command surface for conformance (see
cli-reference.md)
2. Conventions
2.1 RFC 2119 keywords
The key words MUST, MUST NOT, SHOULD, SHOULD NOT, MAY, and REQUIRED are interpreted as described in RFC 2119.
2.2 Check identifiers
Every check has a stable identifier of the form <target>.<category>.<short-name>:
adapter.emit-rules.exactly-one-file
adapter.emit-verbs.intake-present
adapter.watchdog.test-loop-detection
stack-pack.detect.returns-result
stack-pack.skills.has-review-markers
Identifiers MUST NOT be renamed once published. Identifiers MAY be added in minor spec versions.
2.3 Checklist notation
Each check in this spec uses:
□ <check-id> — <description>
Method/area: <what's tested>
Required: MUST | SHOULD
Pass when: <pass criterion>
Fail when: <fail criterion>
3. Conformance model
3.1 Levels are capability outcomes, not a ranking
A conformance level describes what a target CLI can physically support, not a
grade of adapter quality. An adapter declares the highest level its target CLI can
honestly back: the level is a consequence of the platform's primitives, not a
score one adapter "wins" over another. aidokit's three first-party adapters are
equal siblings (ADR-0016, ADR-0031); they differ in declared level only because
their target CLIs differ in capability, not in care or completeness.
| Level | Capability the target CLI must expose |
|---|---|
| Minimum | Targets implementing only the MUST requirements from the relevant contract |
| Standard | Minimum + all SHOULD requirements |
| Strict | Standard + all MAY requirements promoted to expected |
3.2 Per-adapter capability matrix
The level each first-party adapter declares follows directly from whether its
target CLI exposes deterministic hooks (PreToolUse / PostToolUse) and output
styles. Where those primitives are absent, the equivalent guidance ships at the
prompt level instead — which is honest, not lesser. Adapters are listed
alphabetically.
| Adapter | Hooks | Output styles | Declared level | Why |
|---|---|---|---|---|
codex |
No | No (prompt-level only) | Minimum | Codex CLI exposes neither hooks nor output styles, so watchdog/styling are prompt-level |
copilot |
No | No | Minimum | GitHub Copilot CLI exposes neither hooks nor output styles |
claude-code |
Yes | Yes | Strict | Claude Code exposes deterministic hooks and output styles, enabling enforced guarantees |
This asymmetry is honest and intended to remain: it reflects target-CLI capability, not a quality difference between adapters. The per-level checklists in §4 below are the precise expression of these capability outcomes.
3.3 Declaration vs verification
Every target declares a conformance level in its manifest (AdapterManifest.conformance or StackPackManifest.conformance). The harness verifies the claim by running the level's checks against the target's implementation.
Declared > Verified is a conformance violation. The package CI MUST fail. Declared < Verified is acceptable (over-conforming). The harness reports the actual level.
3.4 Inheritance
- Standard includes all Minimum checks
- Strict includes all Standard checks
A pass at level X means the target passed every check at every level ≤ X.
3.5 Per-method vs holistic
Checks fall into two categories:
- Per-method checks — verify a single contract method's output (e.g.,
emitVerbsreturns 3 verb files) - Holistic checks — verify cross-method invariants (e.g., emitted role frontmatter MUST reference real skill ids from
emitSkills)
Both run in the same harness pass.
4. Adapter conformance — per-level checklists
4.1 Minimum conformance — adapter
These checks MUST all pass for an adapter to claim Minimum conformance.
4.1.1 Manifest checks
□ adapter.manifest.declared-level
Method/area: manifest
Required: MUST
Pass when: manifest.conformance is one of 'minimum' | 'standard' | 'strict'
Fail when: missing or invalid value
□ adapter.manifest.spec-version
Method/area: manifest
Required: MUST
Pass when: manifest.specVersion === '2.0'
Fail when: any other value or missing
□ adapter.manifest.capabilities-declared
Method/area: manifest.capabilityDeclarations
Required: MUST
Pass when: writesPaths and runsShellCommands fields are present arrays
Fail when: missing or non-array
□ adapter.manifest.cli-target
Method/area: manifest
Required: MUST
Pass when: cliTarget is a non-empty string with version spec
Fail when: empty or missing
4.1.2 Agent rules file checks
□ adapter.emit-rules.exactly-one-file
Method/area: emitAgentRulesFile
Required: MUST
Pass when: returns a single EmittedFile object (not array, not null)
Fail when: returns anything else
□ adapter.emit-rules.owner-is-adapter
Method/area: emitAgentRulesFile
Required: MUST
Pass when: returned file has owner: 'adapter'
Fail when: any other owner
□ adapter.emit-rules.references-source-of-truth-chain
Method/area: emitAgentRulesFile
Required: MUST
Pass when: file content includes the 10-level source-of-truth chain
Fail when: chain missing or incomplete
□ adapter.emit-rules.includes-validation-commands
Method/area: emitAgentRulesFile
Required: MUST
Pass when: file content references at least one validation command
Fail when: no validation commands referenced
4.1.3 Engine config checks
□ adapter.emit-engine.paths-under-engine-dir
Method/area: emitEngineConfig
Required: MUST
Pass when: every returned EmittedFile.path is under the adapter's declared engine directory
Fail when: any file outside the engine directory
□ adapter.emit-engine.all-paths-relative
Method/area: emitEngineConfig
Required: MUST
Pass when: every path is relative (no leading /, no '..' segments)
Fail when: any absolute path or upward traversal
4.1.4 Verb checks
□ adapter.emit-verbs.intake-present
Method/area: emitVerbs
Required: MUST
Pass when: output contains a file implementing the 'intake' verb
Fail when: missing intake implementation
□ adapter.emit-verbs.implement-task-present
Method/area: emitVerbs
Required: MUST
Pass when: output contains a file implementing the 'implement-task' verb
Fail when: missing
□ adapter.emit-verbs.orchestrator-next-present
Method/area: emitVerbs
Required: MUST
Pass when: output contains a file implementing the 'orchestrator-next' verb
Fail when: missing
□ adapter.emit-verbs.intake-stops-before-code
Method/area: emitVerbs (intake)
Required: MUST
Pass when: intake verb content references the hard-stop-before-code requirement
Fail when: no stop directive present
4.1.5 Role checks
□ adapter.emit-roles.six-required-roles-present
Method/area: emitRoles
Required: MUST
Pass when: output contains role files for: researcher, architect, planner,
builder, tester-reviewer, maintainer
Fail when: any of the six is missing
□ adapter.emit-roles.builder-scope-limited
Method/area: emitRoles (builder)
Required: MUST
Pass when: builder role declares scope-limited-write capability profile
Fail when: builder runs with broader or narrower profile
□ adapter.emit-roles.read-only-roles-correct
Method/area: emitRoles
Required: MUST
Pass when: researcher, architect, planner, tester-reviewer declare read-only profile
Fail when: any of those roles can write
□ adapter.emit-roles.seventh-role
Method/area: emitRoles (frontend-browser-tester)
Required: MAY (Minimum), SHOULD (Standard), MUST (Strict)
Pass when: at Strict, frontend-browser-tester role file is present
Skip when: at Minimum and role is absent
4.1.6 Skill checks
□ adapter.emit-skills.required-skills-emitted
Method/area: emitSkills
Required: MUST
Pass when: every skill in input with required:true appears in output
Fail when: any required skill missing
□ adapter.emit-skills.preserves-triggers
Method/area: emitSkills
Required: MUST
Pass when: emitted skills preserve autoLoadTriggers and preloadedBy from input
Fail when: metadata lost
□ adapter.emit-skills.stack-pack-skills-marked
Method/area: emitSkills
Required: SHOULD
Pass when: stack-pack-contributed skills are visibly marked (header or path convention)
Warn when: indistinguishable from core skills
4.1.7 Watchdog checks
□ adapter.watchdog.test-loop-detection
Method/area: emitWatchdog
Required: MUST
Pass when: watchdog includes mechanism to stop after 3 consecutive test failures
Fail when: no such mechanism
□ adapter.watchdog.command-loop-detection
Method/area: emitWatchdog
Required: MUST
Pass when: watchdog includes mechanism to stop after 5 same-command failures
Fail when: no such mechanism
□ adapter.watchdog.file-edit-counter
Method/area: emitWatchdog
Required: MUST
Pass when: watchdog tracks per-task file-edit counts and warns at threshold
Fail when: no edit tracking
□ adapter.watchdog.repair-cap
Method/area: emitWatchdog
Required: MUST
Pass when: watchdog enforces max 2 repair attempts per task
Fail when: no cap
□ adapter.watchdog.out-of-scope-block
Method/area: emitWatchdog
Required: MUST
Pass when: watchdog blocks writes to files outside the active task's in-scope list
Fail when: not enforced
□ adapter.watchdog.version-bump-gate
Method/area: emitWatchdog
Required: MUST
Pass when: watchdog blocks version bumps without explicit approval flag
Fail when: not enforced
□ adapter.watchdog.tracked-deletion-gate
Method/area: emitWatchdog
Required: MUST
Pass when: watchdog blocks deletion of tracked files without approval
Fail when: not enforced
□ adapter.watchdog.blocker-emitter
Method/area: emitWatchdog
Required: MUST
Pass when: a watchdog stop emits a blocker report under agent-artifacts/blockers/
Fail when: no blocker emission
□ adapter.watchdog.monitoring-stream
Method/area: emitWatchdog
Required: SHOULD (Standard), MUST (Strict)
Pass when: hook writes events.jsonl under agent-artifacts/monitoring/
Skip when: at Minimum and feature absent
4.1.8 Post-install checks
□ adapter.post-install.executable-scripts-marked
Method/area: emitWatchdog / postInstall
Required: MUST
Pass when: every emitted .mjs / .sh file is marked mode: 0o755 OR a chmod command is in postInstall
Fail when: scripts emitted without execute permissions
□ adapter.post-install.beads-init-when-enabled
Method/area: postInstall
Required: SHOULD
Pass when: if ctx.beadsEnabled, postInstall includes a bd init command
Warn when: Beads enabled but not initialized
4.1.9 Determinism checks
□ adapter.determinism.repeated-invocation-stable
Method/area: all emit methods
Required: MUST
Pass when: invoking the same method twice with identical ctx produces byte-identical output
Fail when: outputs differ
4.2 Standard conformance — adapter
Adds to Minimum:
□ adapter.mcp.install-implemented
Method/area: installMCP
Required: SHOULD (Minimum), MUST (Standard)
Pass when: returns valid ShellCommand[] for the test MCP fixture
Fail when: throws or returns invalid commands
□ adapter.mcp.remove-implemented
Method/area: removeMCP
Required: SHOULD (Minimum), MUST (Standard)
Pass when: returns valid removal commands; idempotent (no error if MCP absent)
Fail when: throws or fails on absent MCP
□ adapter.mcp.list-implemented
Method/area: listInstalledMCPs
Required: SHOULD (Minimum), MUST (Standard)
Pass when: returns an array (possibly empty)
Fail when: throws or returns non-array
□ adapter.sync.pre-sync-implemented
Method/area: preSync
Required: SHOULD (Minimum), MUST (Standard)
Pass when: returns a valid SyncPlan
Fail when: throws or returns malformed plan
□ adapter.sync.classifies-all-files
Method/area: preSync
Required: MUST (Standard)
Pass when: every file in the would-emit set appears in exactly one of toCreate/toUpdate/toDelete/toSkip
Fail when: any file unclassified or duplicated
□ adapter.doctor.implemented
Method/area: doctor
Required: SHOULD (Minimum), MUST (Standard)
Pass when: returns DoctorCheck[]; runs all checks (no short-circuit)
Fail when: throws or short-circuits on first failure
□ adapter.doctor.verifies-cli-installation
Method/area: doctor
Required: MUST (Standard)
Pass when: doctor checks target CLI is installed and version matches manifest.cliTarget
Fail when: absent
□ adapter.doctor.verifies-required-files
Method/area: doctor
Required: MUST (Standard)
Pass when: doctor reports on presence of expected engine directory contents
Fail when: absent
4.3 Strict conformance — adapter
Adds to Standard:
□ adapter.emit-roles.seventh-role-strict
Method/area: emitRoles
Required: MUST (Strict)
Pass when: frontend-browser-tester role is emitted
Fail when: missing
□ adapter.output-styles.emitted
Method/area: emitOutputStyles
Required: SHOULD (Standard), MUST (Strict)
Pass when: returns at least one output-style file
Fail when: returns empty array at Strict
□ adapter.watchdog.monitoring-stream-strict
Method/area: emitWatchdog
Required: MUST (Strict)
Pass when: monitoring stream writes structured events for every workflow stage transition
Fail when: incomplete coverage
□ adapter.doctor.runs-conformance-subset
Method/area: doctor
Required: SHOULD (Strict)
Pass when: doctor invokes a subset of the conformance harness
Warn when: absent
4.4 Adapter conformance summary table
| Level | Total MUST checks | Total SHOULD checks |
|---|---|---|
| Minimum | ~24 | ~3 |
| Standard | ~32 | ~6 |
| Strict | ~36 | ~8 |
Exact counts depend on spec evolution; the harness is authoritative.
5. Stack-pack conformance — per-level checklists
5.1 Minimum conformance — stack pack
5.1.1 Manifest checks
□ stack-pack.manifest.declared-level
Required: MUST
Pass when: manifest.conformance is valid
□ stack-pack.manifest.languages-non-empty
Required: MUST
Pass when: manifest.languages is a non-empty array
□ stack-pack.manifest.spec-version
Required: MUST
Pass when: manifest.specVersion === '2.0'
5.1.2 Detection checks
□ stack-pack.detect.returns-result
Method/area: detect
Required: MUST
Pass when: returns a DetectionResult on every invocation
Fail when: throws
□ stack-pack.detect.confidence-categorical
Method/area: detect
Required: MUST
Pass when: result.confidence is one of 'high' | 'medium' | 'low' | 'none'
Fail when: any other value
□ stack-pack.detect.no-shell-calls
Method/area: detect
Required: MUST
Pass when: detection uses only DetectContext methods (verified via harness sandbox)
Fail when: shell execution detected
□ stack-pack.detect.handles-missing-files
Method/area: detect
Required: MUST
Pass when: returns matched:false on a project with no relevant signals
Fail when: throws on absent files
5.1.3 Skill checks
□ stack-pack.skills.at-least-one-skill
Method/area: suggestSkills
Required: MUST
Pass when: returns at least one SkillTemplate
Fail when: returns empty array
□ stack-pack.skills.source-marked
Method/area: suggestSkills
Required: MUST
Pass when: every skill has source: 'stack-pack' and sourcePackId set
Fail when: any skill missing these fields
□ stack-pack.skills.has-review-markers
Method/area: suggestSkills
Required: MUST
Pass when: opinionated skills contain <REVIEW REQUIRED> markers (heuristic check)
Fail when: opinionated language without markers detected
□ stack-pack.skills.no-secrets
Method/area: suggestSkills
Required: MUST
Pass when: skill content contains no values matching common secret patterns
Fail when: API key / token / password patterns detected
□ stack-pack.skills.no-hard-coded-paths
Method/area: suggestSkills
Required: SHOULD
Pass when: skill content does not contain absolute paths or user-home references
Warn when: detected
5.1.4 Determinism checks
□ stack-pack.determinism.repeated-invocation-stable
Method/area: all methods
Required: MUST
Pass when: repeated invocation with identical ctx produces identical output
Fail when: outputs differ
5.2 Standard conformance — stack pack
Adds to Minimum:
□ stack-pack.mcp.references-catalog-ids
Method/area: suggestMCPs
Required: MUST (Standard)
Pass when: every returned id exists in the current MCP catalog
Fail when: any id not in catalog
□ stack-pack.mcp.no-sensitive-suggestions
Method/area: suggestMCPs
Required: MUST
Pass when: no returned id has securitySensitive: true in the catalog
Fail when: any sensitive id returned
□ stack-pack.validation.commands-well-formed
Method/area: suggestValidationCommands
Required: MUST (Standard)
Pass when: every returned command has valid id, label, command, kind
Fail when: missing fields
□ stack-pack.validation.ci-safety-marked
Method/area: suggestValidationCommands
Required: MUST
Pass when: ciSafe field is set on every command
Fail when: missing
□ stack-pack.validation.respects-project-scripts
Method/area: suggestValidationCommands
Required: SHOULD
Pass when: when test fixture has a package.json with matching scripts, commands use 'npm run <script>'
Warn when: pack reinvents commands that exist as scripts
5.3 Strict conformance — stack pack
Adds to Standard:
□ stack-pack.architecture.pattern-implemented
Method/area: defaultArchitecturePattern
Required: SHOULD (Strict)
Pass when: method exists and returns a valid ArchitecturePatternHint
Skip when: method is absent (allowed by spec)
□ stack-pack.architecture.suggested-layout-non-empty
Method/area: defaultArchitecturePattern
Required: MUST when method exists
Pass when: suggestedLayout has at least one entry
Fail when: empty when method is implemented
□ stack-pack.composability.composes-with-declared
Method/area: manifest.composesWith
Required: SHOULD (Strict)
Pass when: manifest declares known composable packs (informational)
Warn when: missing for packs that commonly compose
5.4 Stack pack conformance summary table
| Level | Total MUST checks | Total SHOULD checks |
|---|---|---|
| Minimum | ~11 | ~2 |
| Standard | ~16 | ~3 |
| Strict | ~18 | ~4 |
6. The conformance harness
6.1 Public API
Exported from @aidokit/core:
import type { Adapter, StackPack, ConformanceLevel, ConformanceReport } from '@aidokit/core';
export async function runAdapterConformance(
adapter: Adapter,
level: ConformanceLevel,
options?: ConformanceOptions,
): Promise<ConformanceReport>;
export async function runStackPackConformance(
pack: StackPack,
level: ConformanceLevel,
options?: ConformanceOptions,
): Promise<ConformanceReport>;
export interface ConformanceOptions {
/** Override fixture for testing. */
fixture?: ProjectContext;
/** Subset of check ids to run (default: all at the level). */
onlyChecks?: string[];
/** Skip these check ids. */
skipChecks?: string[];
/** Treat SHOULD failures as warnings (default) or failures. */
strictSoft?: boolean;
}
6.2 Harness flow
1. Identify target type (adapter | stack-pack)
2. Resolve check set for declared level
3. Filter by onlyChecks / skipChecks
4. Build synthetic ProjectContext fixtures (or use override)
5. For each check:
a. Invoke the target method(s) the check covers
b. Apply pass/fail criterion
c. Record ConformanceResult with evidence
6. Aggregate into ConformanceReport
7. Return
6.3 Failure isolation
A check's failure MUST NOT prevent subsequent checks from running. The harness collects all results before returning.
6.4 Performance budget
A full conformance run (~36 adapter checks or ~18 stack-pack checks) MUST complete in under 30 seconds on a developer machine, exclusive of any external CLI invocations.
7. Test fixtures
7.1 Why fixtures
The harness needs synthetic ProjectContext (or DetectContext) objects that exercise different scenarios without requiring real projects. Fixtures are deterministic, version-controlled, and shared across all targets.
7.2 Standard fixtures
Each fixture is a named ProjectContext instance in @aidokit/core/fixtures:
| Fixture id | Description |
|---|---|
greenfield-node-ts |
New Node.js + TypeScript project, no source code yet |
brownfield-node-ts |
Existing Node-TS project with README and prior PRD |
multi-stack-fullstack |
Node-TS + React + Postgres composition |
python-django |
Python + Django brownfield |
go-cli |
Go CLI project |
monorepo-mixed |
Monorepo with TS frontend + Python backend |
minimum-conformance |
Project targeting minimum conformance |
strict-conformance |
Project targeting strict conformance |
Each fixture has a corresponding DetectContext view derived from its file tree.
7.3 Fixture extensibility
Targets MAY contribute additional fixtures via the ConformanceOptions.fixture override. The standard fixtures cover the v1.0 spec; new fixtures appear with new spec versions.
7.4 Determinism
Fixtures MUST be deterministic: re-running the same harness against the same target with the same fixture produces identical results. No network access; no clock-dependent fields; no random data.
8. Reporting format
8.1 ConformanceResult
export interface ConformanceResult {
/** Stable check identifier (see §2.2). */
id: string;
/** Human-readable description of what was tested. */
description: string;
/** Which level this check applies to. */
level: ConformanceLevel;
/** MUST | SHOULD | MAY. */
required: 'MUST' | 'SHOULD' | 'MAY';
/** Outcome. */
status: 'pass' | 'fail' | 'skip' | 'warn';
/** Detail message. */
message: string;
/** Evidence for failures. */
evidence?: {
method?: string;
expected?: unknown;
actual?: unknown;
fixture?: string;
};
/** Suggested fix (optional). */
remediation?: string;
/** Spec reference (filename + section). */
specRef?: string;
}
8.2 ConformanceReport
export interface ConformanceReport {
/** Whether this was an adapter or stack pack run. */
target: 'adapter' | 'stack-pack';
/** Name of the target (manifest.name). */
targetName: string;
/** Target's declared conformance level. */
declaredLevel: ConformanceLevel;
/** Level the harness verified against. */
testedLevel: ConformanceLevel;
/** All check results, in execution order. */
results: ConformanceResult[];
/** Summary counts. */
summary: {
total: number;
passed: number;
failed: number;
skipped: number;
warned: number;
};
/** Overall pass/fail at testedLevel. */
overall: 'pass' | 'fail';
/** Timestamp of the run. */
ranAt: string;
/** Versions of relevant components. */
versions: {
aidoCore: string;
specVersion: string;
targetVersion: string;
};
}
8.3 Human-readable formatter
The harness ships a default formatter:
Conformance Report: claude-code@0.1.0
Declared: Strict · Tested: Strict
Spec: v2.0 · Harness: @aidokit/core@1.2.0
Summary: 32 passed · 0 failed · 4 skipped · 2 warned
Overall: PASS
Skipped:
· adapter.emit-roles.seventh-role-strict — only required at Strict
· adapter.output-styles.emitted — only required at Strict
· (2 more)
Warnings:
⚠ adapter.post-install.beads-init-when-enabled
Beads is enabled in fixture but no `bd init` in postInstall.
Remediation: add bd init to postInstall when ctx.beadsEnabled.
8.4 Machine-readable formatter
--format=json (when invoked via CLI) emits the ConformanceReport object directly.
9. CI integration
9.1 Adapter package CI
Every adapter package MUST include a test:conformance script:
{
"scripts": {
"test:conformance": "vitest run conformance.test.ts"
}
}
And a conformance test file (packages/adapter-*/test/conformance.test.ts):
import { test, expect } from 'vitest';
import { runAdapterConformance } from '@aidokit/core';
import { adapter } from '../src/index.js';
test(`${adapter.manifest.name} conforms to declared level`, async () => {
const report = await runAdapterConformance(adapter, adapter.manifest.conformance);
if (report.overall !== 'pass') {
const failures = report.results.filter((r) => r.status === 'fail');
console.error(JSON.stringify(failures, null, 2));
}
expect(report.overall).toBe('pass');
});
9.2 Stack pack package CI
Mirror the above for stack packs:
import { runStackPackConformance } from '@aidokit/core';
import { stackPack } from '../src/index.js';
test(`${stackPack.manifest.name} conforms to declared level`, async () => {
const report = await runStackPackConformance(stackPack, stackPack.manifest.conformance);
expect(report.overall).toBe('pass');
});
9.3 Required gates
The package's CI workflow MUST run pnpm test:conformance and fail the build on non-zero exit. Publishing a package that fails conformance is prohibited.
9.4 Reproducibility
Conformance reports MUST be reproducible: a given target version + harness version + spec version always produces the same report (modulo timestamps).
10. aidokit doctor integration
10.1 Subset run
aidokit doctor re-runs a subset of the harness on the user's installed adapter to detect:
- Adapter behavior that has drifted from its declared conformance
- Adapter version mismatches
- Engine directory tampering
The subset (~10–12 checks) is fast (<5 seconds) and focuses on structural integrity.
10.2 Drift detection
aidokit doctor compares the live conformance report to the report captured at install time (stored in .aido/conformance-snapshot.json). Differences are reported.
10.3 Exit semantics
- All checks pass → exit 0
- Any MUST check fails → exit 13 (CONFORMANCE_FAILED per
cli-reference.md§10.2) - Only SHOULD checks warn → exit 0 with warnings (unless
--strict)
10.4 What aidokit doctor doesn't do
aidokit doctor does NOT:
- Run the full conformance harness (too slow for user-time)
- Modify project state
- Re-execute adapter postInstall
For a full conformance check, run the package CI in the adapter's repo.
11. Project-level conformance
11.1 Declaration
A user's project declares its target conformance level in <projectRoot>/.aido/adapter.md:
---
adapter: claude-code
conformanceTarget: standard
specVersion: '2.0'
---
11.2 Constraint
The project's conformanceTarget MUST be <= the installed adapter's manifest.conformance. A project at Strict requires an adapter declaring Strict.
aidokit init enforces this at install time:
✘ Adapter 'codex' declares Minimum conformance; cannot target Strict.
Either choose a different adapter or lower your target.
11.3 Inheritance
The project conformance level applies to every operation: file emission verification, doctor checks, MCP suggestion strictness.
12. Trust signals and marketplace
12.1 Verified badge
The post-1.0 marketplace (v2.0 per ROADMAP.md) grants a "Verified" badge to packages that:
- Pass conformance in CI on every published version
- Make conformance reports publicly accessible (e.g., in repo as
conformance-report.json) - Sign packages with npm provenance
12.2 Trust tiers
| Tier | Criteria |
|---|---|
| Official | @aidokit/* scope; maintained by core team |
| Verified | Passes conformance CI; signed; active maintainer |
| Community | Published; follows naming convention; no other signals |
| Unverified | Matches naming but has known issues (deprecated, abandoned, etc.) |
12.3 Display in aidokit search
$ aido search adapter
@aidokit/adapter-codex official @ Minimum conformance (verified)
@aidokit/adapter-copilot official @ Minimum conformance (verified)
@aidokit/adapter-claude-code official @ Strict conformance (verified)
aidokit-adapter-aider community @ claimed Standard (unverified)
@acme/aidokit-adapter-mytool verified @ Minimum conformance
13. Failure modes and recovery
13.1 Failure during aidokit init
If aidokit init invokes a conformance subset on the adapter being installed and it fails:
✘ Adapter conformance check failed: claude-code@0.1.0 (Strict claimed)
Failing checks:
· adapter.emit-roles.builder-scope-limited
Builder role declared default permissionMode, expected with PreToolUse hook
Try: pnpm update @aidokit/adapter-claude-code
Exit code: 13 (CONFORMANCE_FAILED).
13.2 Failure during aidokit doctor
✘ Conformance drift detected
· adapter.emit-skills.required-skills-emitted
Skill 'beads-queries' missing from .claude/skills/
Try: aido sync
13.3 Recovery strategies
| Failure | Recovery |
|---|---|
| Adapter version mismatch | aidokit sync to re-emit |
| Skill file deleted | aidokit sync --skills |
| Hook scripts lost execute permission | aidokit sync --scripts or chmod +x manually |
| Adapter no longer claims declared level | aidokit doctor --strict to inspect; upgrade or replace adapter |
| MCP scoping divergence | aido mcp scope <id> --set-roles <list> |
14. Versioning and evolution
14.1 Check stability
Once published, a check id is stable. The pass/fail criterion MAY be refined (with care) but the id and its semantic intent MUST NOT change.
14.2 Adding checks
New checks may be added in minor spec versions. Existing targets retain their declared conformance (the new check applies to the next major spec version).
14.3 Removing checks
A check MAY be marked deprecated (excluded from current runs) but MUST remain in the spec for historical reference.
14.4 Level redefinition
Promoting a check from SHOULD to MUST is a breaking change (major bump). Demoting is non-breaking.
15. Open questions
| Question | Resolution target |
|---|---|
Should --strict doctor mode be the default in CI environments? |
Spec 1.0 freeze |
| Whether to add a Conformance ID badge SVG generator | v2.0 marketplace |
| How to handle conformance for adapters that target older spec versions | Spec 1.1 |
Whether to ship a aidokit conformance-report standalone CLI command |
v1.0 |
| Whether targets should declare which checks they intentionally skip (with rationale) | Spec 1.1 |
Appendix A: Synthetic ProjectContext fixtures
A.1 greenfield-node-ts
{
projectRoot: '/tmp/aidokit-conformance/greenfield-node-ts',
projectName: 'example-app',
stack: {
languages: ['typescript'],
frameworks: [],
},
brownfield: false,
sourceDocs: [],
adapterName: '<provided-by-harness>',
conformance: 'standard',
beadsEnabled: true,
selectedStackPacks: ['node-ts'],
selectedMCPs: ['context7', 'beads-mcp'],
os: 'linux',
arch: 'x64',
installedPrereqs: [
{ id: 'node', version: '20.11.0', installed: true },
{ id: 'git', version: '2.43.0', installed: true },
{ id: 'beads', version: '0.9.0', installed: true },
],
aidoCliVersion: '0.1.0',
specVersion: '2.0',
options: {},
}
A.2 brownfield-node-ts
Same as greenfield-node-ts but with:
{
brownfield: true,
sourceDocs: [
{ path: 'docs/inputs/source-prd.md', kind: 'prd' },
{ path: 'README.md', kind: 'readme' },
],
}
A.3 multi-stack-fullstack
{
// ... same base
selectedStackPacks: ['node-ts', 'react', 'postgres'],
selectedMCPs: ['context7', 'beads-mcp', 'playwright', 'postgres-mcp'],
}
A.4 strict-conformance
{
// ... base
conformance: 'strict',
}
Appendix B: Full check id reference
Generated from this spec; canonical list lives in @aidokit/core/src/conformance/checks.ts.
B.1 Adapter checks (~36)
adapter.manifest.declared-level
adapter.manifest.spec-version
adapter.manifest.capabilities-declared
adapter.manifest.cli-target
adapter.emit-rules.exactly-one-file
adapter.emit-rules.owner-is-adapter
adapter.emit-rules.references-source-of-truth-chain
adapter.emit-rules.includes-validation-commands
adapter.emit-engine.paths-under-engine-dir
adapter.emit-engine.all-paths-relative
adapter.emit-verbs.intake-present
adapter.emit-verbs.implement-task-present
adapter.emit-verbs.orchestrator-next-present
adapter.emit-verbs.intake-stops-before-code
adapter.emit-roles.six-required-roles-present
adapter.emit-roles.builder-scope-limited
adapter.emit-roles.read-only-roles-correct
adapter.emit-roles.seventh-role
adapter.emit-roles.seventh-role-strict
adapter.emit-skills.required-skills-emitted
adapter.emit-skills.preserves-triggers
adapter.emit-skills.stack-pack-skills-marked
adapter.watchdog.test-loop-detection
adapter.watchdog.command-loop-detection
adapter.watchdog.file-edit-counter
adapter.watchdog.repair-cap
adapter.watchdog.out-of-scope-block
adapter.watchdog.version-bump-gate
adapter.watchdog.tracked-deletion-gate
adapter.watchdog.blocker-emitter
adapter.watchdog.monitoring-stream
adapter.watchdog.monitoring-stream-strict
adapter.post-install.executable-scripts-marked
adapter.post-install.beads-init-when-enabled
adapter.determinism.repeated-invocation-stable
adapter.mcp.install-implemented
adapter.mcp.remove-implemented
adapter.mcp.list-implemented
adapter.sync.pre-sync-implemented
adapter.sync.classifies-all-files
adapter.doctor.implemented
adapter.doctor.verifies-cli-installation
adapter.doctor.verifies-required-files
adapter.doctor.runs-conformance-subset
adapter.output-styles.emitted
B.2 Stack pack checks (~18)
stack-pack.manifest.declared-level
stack-pack.manifest.languages-non-empty
stack-pack.manifest.spec-version
stack-pack.detect.returns-result
stack-pack.detect.confidence-categorical
stack-pack.detect.no-shell-calls
stack-pack.detect.handles-missing-files
stack-pack.skills.at-least-one-skill
stack-pack.skills.source-marked
stack-pack.skills.has-review-markers
stack-pack.skills.no-secrets
stack-pack.skills.no-hard-coded-paths
stack-pack.determinism.repeated-invocation-stable
stack-pack.mcp.references-catalog-ids
stack-pack.mcp.no-sensitive-suggestions
stack-pack.validation.commands-well-formed
stack-pack.validation.ci-safety-marked
stack-pack.validation.respects-project-scripts
stack-pack.architecture.pattern-implemented
stack-pack.architecture.suggested-layout-non-empty
stack-pack.composability.composes-with-declared
See also
adapter-contract.md— Adapter interface and per-method MUST/SHOULD/MAYstack-pack-contract.md— Stack-pack interface and per-method requirementscli-reference.md—aidokit doctorcommand surface andCONFORMANCE_FAILEDexit codemcp-catalog.md— MCP catalog (referenced by stack-pack MCP checks)../architecture/decisions/0005-adapter-contract-shape.md../architecture/decisions/0006-stack-pack-contract-shape.md- Upstream
orchestrator-spec.md§18 — Conformance levels in the tool-agnostic spec