ADR 0025: Deterministic known-broken baseline + failure classifier

Summary

aidokit gains a baseline command with two subcommands. aidokit baseline capture --from <file> parses a test-output file into the sorted, de-duplicated set of failing identifiers and writes it to .aidokit/baseline.json. aidokit baseline diff --from <file> parses a new test-output file and classifies each failure as pre-existing (present in the baseline) or new (the classifier), exiting 1 when new failures appear.

aidokit never runs the project's test suite. It parses a file the caller produced — keeping the feature fully deterministic, hermetic, and free of LLM or network calls.

Context

When an agent works on a brownfield project, an existing red test suite makes it impossible to tell whether a change introduced a regression or merely exposed a pre-existing failure. Re-running the suite and eyeballing the output is non-deterministic and burns context. The agent needs a stable "known-broken" snapshot to diff against so it can isolate the failures it is responsible for.

The constraint (CLAUDE.md non-goals): aidokit makes zero LLM calls and runs no daemons. It must also not shell out to run an arbitrary, project-specific test command — that is non-deterministic and a security surface. The deterministic slice is parsing a file the caller already produced.

Decision

1. Core logic in @aidokit/core/baseline.ts

• parseFailures(text): readonly string[] — recognises two line shapes: TAP-ish not ok <n?> - <name> (leading index and - separator stripped, trailing # directive comment dropped) and plain FAIL <name> / FAILED <name> (case-insensitive). Returns a sorted, de-duplicated set.
• captureBaseline(text): Baseline — wraps parseFailures into { failures }.
• diffBaseline(baseline, current): BaselineDiff — partitions the current failing set into preExisting, newFailures, and fixed.

All functions are pure: text in, structured data out. No I/O.

2. CLI command in packages/cli/src/commands/baseline.ts

baseline capture --from <file> writes .aidokit/baseline.json ({ "failures": [...] }, sorted). baseline diff --from <file> loads the baseline, parses the new file, and prints the classification; exit 1 when newFailures is non-empty, else 0. Missing input file or missing baseline raises AidokitError with code BAD_INVOCATION.

3. No new finding codes

The command reuses the existing BAD_INVOCATION error code. No additions to error-codes.md.

Consequences

Positive

• Deterministic and hermetic. Same input file always yields the same baseline and diff. CI-safe.
• No new attack surface. aidokit never executes the project's tests.
• Cheap classifier. The agent gets a clean pre-existing-vs-new split without re-reading raw test logs.

Negative

• Parser coverage is heuristic. Only TAP not ok and FAIL/FAILED lines are recognised. Exotic reporters need pre-normalisation by the caller. Mitigation: the two shapes cover the dominant ecosystems (TAP, Jest/Vitest summary lines); the format can be extended non-breakingly.

Neutral

• .aidokit/baseline.json is a runtime artifact, not emitted by init, so it does not touch the byte-compare dogfood gate.

Alternatives considered

Alternative 1: Run the suite ourselves

Rejected — non-deterministic, project-specific, and a shell-execution surface that violates the zero-side-effects posture.

Alternative 2: Store the baseline in Beads

Rejected for v1.0 — couples a deterministic scaffolder feature to an external runtime. A flat JSON file in .aidokit/ is the smallest thing that works.

Validation

This decision is validated when:

• [x] parseFailures extracts TAP and plain failures, sorted and de-duped.
• [x] baseline capture writes .aidokit/baseline.json.
• [x] baseline diff classifies pre-existing vs new and exits 1 on new.
• [x] Missing input or missing baseline raises BAD_INVOCATION.

References

Internal

• docs/specs/cli-reference.md — baseline command surface.
• ADR-0009 — Commander (command wiring).