docs(openspec): propose linking tests to spec scenarios

Phase 1 for #1967: tests and other evidence point at the scenario they verify with an @spec reference, and a read-only coverage report lists linked, unlinked, and dangling references. Specs keep their format. Refs #1967 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 22:13:19 +08:00 · 2026-09-25 08:25:13 -05:00
4 changed files with 165 additions and 0 deletions
@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-09-25
@@ -0,0 +1,80 @@
+## Context
+
+#1967 names two symptoms with one cause. Specs fill up with test and implementation detail because they are the only artifact that stays live after archive. And nothing links a requirement to the code or tests behind it. The issue asks for careful thinking before anything is built, and leaves three questions open. This design answers them for a first, small step, and parks the rest.
+
+What exists today:
+
+| Piece | State |
+|---|---|
+| Requirement identity | Heading text. Archive matches MODIFIED/REMOVED/RENAMED by normalized name (`normalizeRequirementName`). |
+| Scenario identity | Heading text, but the parsed model keeps only `rawText`, not the name. There is no scenario-level RENAMED. |
+| `**ID**:` metadata lines | The body reader already skips them (`requirement-text.ts`), but nothing reads or checks them. |
+| `specs/<capability>/design.md` | Listed in the `openspec-conventions` project structure ("HOW, optional"). No code reads, writes, or archives it. |
+| `/opsx:verify` scenario coverage | The agent searches for tests that look related. Nothing is recorded. |
+
+## Goals / Non-Goals
+
+**Goals**
+- Let evidence point at the behavior it proves, without changing how specs are written.
+- Make gaps visible: scenarios with no link, and links that no longer point at anything.
+
+**Non-Goals (this change)**
+- Stable IDs in specs.
+- A home for implementation context (see Follow-ups).
+- Gating validate or archive on coverage (#1652 owns that question).
+- Generating tests (#900).
+
+## Decisions
+
+### 1. The unit of linking is the scenario
+
+Scenarios are the testable part of a spec: WHEN/THEN is already a test shape. A requirement's status is the rollup of its scenarios. A reference without a scenario part links the whole requirement. That covers a test that proves the requirement's overall rule rather than one case.
+
+### 2. Links live in the evidence, not in the spec
+
+The test (or a manual-check note, or a lint config) carries the reference. Specs stay clean, which is the point of #1967. The link also sits next to the thing most likely to change, so whoever deletes a test deletes its claim with it.
+
+Rejected: links in the spec. That puts file paths and test names back into requirements, the exact leak the issue describes. Rejected: a separate index file. Nobody updates it, and it drifts from both the specs and the tests.
+
+### 3. Address by heading text, not new IDs
+
+`@spec cli-archive > Archive refuses scenario loss > MODIFIED block drops a scenario`
+
+Heading text is already the identity archive uses. It is readable in a diff and needs nothing new in the spec. Its weakness is that renames break links. The report turns that into a visible finding (a dangling reference) instead of a silent gap. Opaque IDs survive renames, but they add structure to every requirement and scenario. Agents also invent or duplicate them, and they mean nothing to a human reader. That is a lot of cost for an advisory report. If dangling links turn out to be common in practice, IDs can be added later using the `**ID**:` lines the reader already skips. Heading-text references would keep working alongside them.
+
+Matching uses the same normalization archive uses for requirement names, and the same rule for scenario names. The separator is ` > ` because `#` is legal in names (`C#`). A capability path never contains spaces, so the first ` > ` always ends the path. A requirement name that itself contains ` > ` can't be referenced unambiguously, and the report says so rather than guessing.
+
+### 4. A read-only report, advisory only
+
+`openspec spec coverage [spec-id] [--json]` scans git-tracked text files in the project root for `@spec` references. It reads the main specs, which live in the store when one is configured. Then it reports:
+
+| Section | Meaning |
+|---|---|
+| Linked | Scenario, with each `file:line` that references it |
+| Unlinked | Scenario with no reference |
+| Dangling | Reference that matches no current requirement or scenario, with the nearest existing name as a hint |
+
+It exits 0 whatever it finds. "Unlinked" means nothing points here. It does not mean unverified, and the output says so. A project that adopts links gradually should not see red.
+
+### 5. Git-tracked files only
+
+Scanning what git tracks skips build output, dependencies, and anything ignored. There's no include/exclude config to maintain. Outside a git repository, the command reports that it can't scan and exits 0.
+
+## Risks / Tradeoffs
+
+- **Renames break links.** Accepted, and surfaced as dangling references. Watch how often this happens before adding IDs.
+- **A reference is a claim, not proof.** A test can link a scenario and assert nothing useful. The report shows where to look. It does not vouch for the evidence (#1937 covers how a reviewer judges it).
+- **Scan cost on large repos.** The scan is a single pass over tracked files for a fixed marker, the same cost class as `git grep`.
+
+## Follow-ups (not in this change)
+
+1. **Lasting home for implementation context.** Either implement the `specs/<capability>/design.md` the conventions spec already promises, with archive merging a change's relevant design notes into it, or remove the promise. This needs its own proposal, because archive would start writing a second file per capability.
+2. **Verify uses the links.** `/opsx:verify` Scenario Coverage reads the report instead of guessing, and reports linked, unlinked, and dangling separately.
+3. **Verification plan at proposal time.** Once links exist, `tasks.md` guidance can ask which scenarios each test task will link, and name the ones that will be verified manually or not at all.
+4. **Strict mode.** Whether `--strict` should fail on dangling references, once there is data on how noisy they are.
+
+## Open Questions for review
+
+- Is heading-text addressing (Decision 3) acceptable as the first step, or should stable IDs come first?
+- `openspec spec coverage` versus a top-level `openspec coverage`?
+- Should manual verification have its own marker (for example `@spec-manual`), so the report can tell "tested" from "checked by hand"?
@@ -0,0 +1,34 @@
+## Why
+
+Specs are the only artifact that stays live after a change archives. Design and tasks go into the archive with the change, and nothing links them back to the requirements they support. So when a detail matters later, the spec is the only place it can survive. That is why test procedures and framework settings end up inside requirements (#1967).
+
+The same gap hides verification. After archive, nothing records how a requirement is verified, or whether it is. `/opsx:verify` guesses scenario coverage by searching for tests that look related, and the answer is gone once the session ends.
+
+Tests already live as long as specs do. If a test can say which scenario it verifies, the test becomes the lasting home for "how is this proven." The spec can then stay a behavior contract.
+
+## What Changes
+
+- A test, or any other tracked text file, can link to a scenario with a one-line reference in a comment:
+  `@spec <capability-path> > <Requirement name> > <Scenario name>`. Dropping the scenario part links the whole requirement.
+- A new read-only command, `openspec spec coverage [spec-id]`, lists every scenario in the main specs with the files and lines that link to it. It also lists scenarios nothing links to, and references that no longer match any scenario (dangling).
+- The report is advisory. It never changes the exit code and never blocks validate or archive.
+- Specs do not change format. No IDs, no new sections, no new metadata.
+
+This is phase 1 of #1967: links from evidence to specs. Where implementation context should live, and a verification plan at proposal time, are separate follow-ups listed in `design.md`.
+
+## Capabilities
+
+### New Capabilities
+
+- `spec-traceability`: scenario references in project files, and a coverage report built from them.
+
+### Modified Capabilities
+
+_None._
+
+## Impact
+
+- **Public CLI:** one new subcommand, `openspec spec coverage`, with `--json`. No change to existing commands, flags, or exit codes.
+- **Spec authors:** nothing new to write. Specs look the same.
+- **Test authors:** optional one-line comments. A project that adds none sees every scenario reported as unlinked, and nothing fails.
+- **Implementation:** a read-only scanner under `src/core/`, a subcommand in `src/commands/spec.ts`, docs in `docs-lab/reference/cli.md`, tests, and a minor changeset. No new dependency.
@@ -0,0 +1,49 @@
+## Purpose
+
+Lets tests and other evidence point at the spec scenarios they verify, so a project can see which behavior is linked to proof and which is not, without adding structure to specs.
+
+## ADDED Requirements
+
+### Requirement: Scenario references in project files
+The system SHALL recognize a reference of the form `@spec <capability-path> > <Requirement name>` optionally followed by ` > <Scenario name>`, on any line of a git-tracked text file. Names SHALL match using the same normalization archive applies to requirement headings.
+
+#### Scenario: Reference to a scenario
+- **WHEN** a test file contains `// @spec cli-archive > Archive refuses scenario loss > MODIFIED block drops a scenario`
+- **THEN** the reference links that file and line to that scenario
+
+#### Scenario: Reference to a whole requirement
+- **WHEN** a reference names a capability and a requirement but no scenario
+- **THEN** the reference links to the requirement as a whole
+
+#### Scenario: Name containing a hash
+- **WHEN** a requirement is named `Parse C# projects`
+- **THEN** `@spec parsers > Parse C# projects` links to it
+
+### Requirement: Coverage report
+`openspec spec coverage [spec-id]` SHALL list every scenario in the main specs, or in the given spec only, as linked (with each referencing `file:line`) or unlinked. It SHALL read main specs from the resolved OpenSpec root, including a configured store. With `--json`, it SHALL emit the same data as a JSON document.
+
+#### Scenario: Linked and unlinked scenarios
+- **WHEN** one scenario of a requirement has a reference and another has none
+- **THEN** the report lists the first as linked with its `file:line` and the second as unlinked
+
+#### Scenario: Requirement-level reference
+- **WHEN** a reference names only a requirement
+- **THEN** the report shows that link on the requirement, and does not count its scenarios as linked
+
+### Requirement: Dangling references are reported
+The report SHALL list every reference that matches no current requirement or scenario as dangling. It SHALL give the file, the line, and the nearest existing name when one is close.
+
+#### Scenario: Requirement renamed after the test was written
+- **WHEN** a test references `cli-archive > Old name` and the requirement is now `New name`
+- **THEN** the report lists the reference as dangling, with `New name` as the nearest match
+
+### Requirement: The report is advisory
+`openspec spec coverage` SHALL exit with status 0 whatever it finds, including unlinked scenarios, dangling references, or no git repository to scan. It SHALL NOT change the behavior or exit code of `validate` or `archive`.
+
+#### Scenario: Nothing is linked
+- **WHEN** a project has no `@spec` references
+- **THEN** every scenario is reported as unlinked and the command exits 0
+
+#### Scenario: Outside a git repository
+- **WHEN** the command runs where no git repository can be found
+- **THEN** it reports that it cannot scan for references and exits 0