6.3 — CI review driver

Let your CI pipeline resolve VibeMap kanban reviews automatically. When your agent submits a criterion for review (with the commit, the changed files and the tests that prove it), your test workflow can post its outcome and the criterion moves to passed or failed — no human click required. Human resolution in the web app keeps working as the fallback, and agents can never resolve their own reviews.

1. Create a CI token

A CI token is a personal access token whose single scope is ci_review.

In the app, go to Account → Developer, enter a name, set the scope selector to CI review, and click Generate Key. The scope of every existing key is shown in the Scope column of the keys table.

⚠️ Pick the scope before you generate. A key's scope is fixed at creation — there is no way to change it afterwards. An Agent key posted to ci-result is rejected with 403 agents_cannot_self_resolve, which is the gate working as designed: the token that does the work must not be the token that passes it.

You can also mint one with an authenticated request, from a browser session on vibemap.ai:

curl -X POST https://vibemap.ai/api/account/keys \
  -H "Content-Type: application/json" \
  --cookie "$SESSION" \
  -d '{"name": "GitHub Actions CI", "scopes": ["ci_review"]}'

scopes takes exactly one entry — agent or ci_review, never both. The token comes back once, in the token field of the response.

A ci_review token can only post review outcomes. It cannot claim, submit, or edit anything, so it is safe to store as a CI secret. Add it to your repository as the VIBEMAP_CI_TOKEN secret.

2. Name your tests after the criterion they prove

Every acceptance criterion has a stable code such as AC-042 (shown on its card, and never changed once assigned). Start the name of each test that proves it with that code in square brackets:

describe("page title", () => {
  it("[AC-042] rejects an empty title", () => { /* ... */ });
  it("[AC-042] trims surrounding spaces", () => { /* ... */ });
  it("[AC-043] caps the length at 80 characters", () => { /* ... */ });
});

• The code is AC- plus at least three digits, in capitals and square brackets. AC-42, ac-042 and AC-042 without the brackets are not recognised.
• A test that proves several criteria carries several tags: [AC-044] [AC-045] shared helper.
• Tests with no tag are ignored. They never pass or fail a criterion.
• Put the same code in your commits as a trailer so the history is searchable: Spec-Ref: AC-042. The trailer is a convention for people and tooling; VibeMap resolves criteria from the test names, not from commit messages.

3. Report the run with ci-report

Version. ci-report ships in @vibemap.ai/mcp-server from version 2.11.0. Older versions do not have the command. In CI, pin the version so a run never depends on whatever is newest: npx -y @vibemap.ai/[email protected] ci-report ....

After the tests run, post the report with the command that ships in the MCP server package:

VIBEMAP_CI_TOKEN=... npx @vibemap.ai/mcp-server ci-report \
  --junit junit.xml --project "$PROJECT_ID"

It reads the report, groups the tagged tests by criterion and posts them. For each criterion that is in review and mentioned by the run:

If the run says which commit it tested (ci-report always does on GitHub Actions), a criterion is only resolved when that commit is the one the agent submitted. Otherwise it is left in review and listed as sha_mismatch with both shas, so a green run on some other commit can never pass the work. A run that does not state its commit still resolves, and the event is marked sha_unverified.

Criteria the run mentions that are not in review (not built yet, already passed) or do not exist in the project are ignored and listed in the response. A test that exists before its criterion is built can never move a card.

Supported reports: --junit (JUnit XML, which Vitest, Jest, Playwright and most runners can write), --vitest (Vitest or Jest JSON) and --playwright (Playwright JSON). Pass several flags to combine suites. On GitHub Actions the run URL and the tested commit are filled in for you; elsewhere pass --run-url and --sha. --dry-run prints the request without sending it, and --strict fails the step when no test carries a tag.

The command exits 0 when the report was accepted, 1 when the server rejected it or could not be reached, and 2 for a usage problem (missing token, project or report). A failing test does not make ci-report fail: the verdict is data, and your test step decides whether the job is red.

The project id is in the project's URL. If you would rather address one criterion, pass --criterion with its id; the report still resolves every criterion in that project.

4. GitHub Actions example

name: vibemap-review
on:
  pull_request:

jobs:
  test-and-report:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci

      - name: Run tests
        id: tests
        # JUnit output; with Vitest: --reporter=junit --outputFile=junit.xml
        run: npx vitest run --reporter=default --reporter=junit --outputFile.junit=junit.xml
        continue-on-error: true

      - name: Report to VibeMap
        env:
          VIBEMAP_CI_TOKEN: ${{ secrets.VIBEMAP_CI_TOKEN }}
          VIBEMAP_PROJECT_ID: ${{ vars.VIBEMAP_PROJECT_ID }}
        # Pin the version so a run never depends on whatever is newest.
        run: npx -y @vibemap.ai/[email protected] ci-report --junit junit.xml

      - name: Fail the job if tests failed
        if: steps.tests.outcome != 'success'
        run: exit 1

Keep the report step before the final failure step so a red run still reports its failures. The tested commit is the pull request head, and the run URL is the Actions run, both read from the environment.

5. Without the command: the HTTP endpoints

ci-report is a thin client. You can post the same body yourself.

POST /api/mcp/kanban/projects/{projectId}/ci-result resolves every in-review criterion the run mentions:

{
  "outcome": "failed",
  "test_run_url": "https://github.com/you/repo/actions/runs/123",
  "git_sha": "the sha that was tested",
  "per_test": [
    { "name": "[AC-042] rejects an empty title", "ac_codes": ["AC-042"], "outcome": "passed" },
    { "name": "[AC-043] caps the length", "ac_codes": ["AC-043"], "outcome": "failed" }
  ]
}

The response lists what happened: { "resolved": [{ "ac_code", "criterion_id", "outcome", "event" }], "ignored": [{ "ac_code", "reason", "status"? }] }. reason is unknown_ac_code, not_in_review, skipped (every test for it was skipped or marked fixme, so CI proved nothing and it stays in review), sha_mismatch (carries submitted_sha and tested_sha), race (a human resolved it first) or transition_failed. outcome at the top is the run's overall result; each criterion is decided by its own tests.

POST /api/mcp/kanban/criterion/{criterionId}/ci-result takes the same body. With per_test it behaves like the project route (using that criterion's project). Without per_test, outcome is applied to that single criterion, and test_run_url is required.

Limits: per_test must not be empty, and may name at most 500 distinct AC codes (422 too_many_ac_codes); requests are rate-limited per token (300 a minute), the env CI token included. Responses: 200 resolved, 409 race, 403 wrong token type or scope, 404 unknown criterion or project, 422 illegal_transition (single-criterion form, not in review) or 422 invalid_payload. Field-level reference: 6.2 — API reference.

Generated tests are untrusted

VibeMap writes an executable test for each criterion and ships it in your repository. Treat that code like any other code you did not write yourself:

• Review it before you commit it. The generator checks each test for the criterion prefix, a real assertion, and a set of forbidden constructs, but that check is a quality gate, not a security boundary. No static check can prove arbitrary code safe.
• Run it in a CI job with no production secrets. Give the job only BASE_URL and the test credentials the tests read from TEST_ variables. Do not expose deploy keys, service-role keys, cloud credentials or any other secret to that job.
• Restrict its network access. The job should reach the app under test and nothing else. Generated tests only call paths on BASE_URL; a test that tries another host is a defect, and egress rules turn it into a failed run instead of a leak.
• Point BASE_URL at a throwaway environment, not production, and use test accounts whose data you can lose.

A criterion whose generated test is marked as needing review is written as a skipped test. CI reports it as skipped, so the criterion stays in review until someone writes the real assertion and deletes the marker line.

What happens on failure

A failed criterion keeps the failing test names and the test_run_url in its event. The assigned agent can then re-queue it (POST .../rework, documented in 6.2); the retry automatically carries that feedback. Or a human can send it back from the board. An agent that isn't the assignee gets 403 not_assigned_agent.

Build evidence

Every submission records what the agent says it built: the commit (git_sha, base_sha), the changed files and the tests that prove each criterion. When CI resolves the review, VibeMap:

• marks the criterion CI-verified (verification_kind: ci). A human pass from the board is recorded as manual and needs a note; only CI marks work verified;
• links the criterion to each file in that submission's own file list (deleted files excluded), with confidence 1.0;
• caches the latest evidence on the criterion.

GET /api/mcp/kanban/criterion/{criterionId}/evidence?limit=20 returns the history, newest first, one entry per submission attempt:

{
  "criterion_id": "...", "ac_code": "AC-042", "status": "passed", "verification_kind": "ci",
  "attempts": [{
    "sha": "...", "base_sha": "...",
    "files": [{ "path": "lib/title.ts", "status": "added" }],
    "tests": [{ "file": "tests/title.test.ts", "name": "[AC-042] rejects an empty title", "ac_codes": ["AC-042"] }],
    "outcome": "passed", "test_run_url": "...", "verification_kind": "ci",
    "resolved_by_kind": "ci", "failing_tests": [], "manual_note": null,
    "submitted_at": "...", "at": "..."
  }]
}

outcome is passed, failed, pending (in review now) or superseded (resubmitted, or sent back by a spec change, before a verdict). The history is read from the event log, so it is complete even when the cached copy is not. limit is 1 to 50 (default 20).

Where to go next

• Endpoint reference → 6.2 — Kanban tracker API reference
• The agent loop this closes → 6.1 — Kanban tracker user guide