Files
substrate/.agents/skills/post-draft-review/SKILL.md
T
Benjamin Elder aa17b3c297 post-draft-review skill for AI *assisted* PR reviews (#870)
This is a skill I've been iterating on, first with a few coworkers' PRs
only, and then progressively more broadly. It gives pretty decent
results with frontier models, but I think we need to continue refining
it.

The basic idea is that this creates or updates draft PR comments to add
AI findings, categorized by severity, and *clearly denoted* as coming
from an agent. It is instructed to bias towards brevity as these models
tend to write massive comments. That still needs more tweaking.

Reviewers are expected to leverage this skill to suggest draft comments
which they then review and delete as needed, and finally submit an
overall review with their own top-level comments.

It does NOT do things like "summarize what the PR does" ... PR authors
already do this, and adding that information to the PR thread is not
helpful. Anyone interested can ask their agent to summarize the PR for
them locally.

What it does do, is report possible bugs, for a reviewer to vet and then
post as they see fit.

NOTE: **this should be squash merged**


> It's a good idea to open an issue first for discussion.

- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
2026-09-21 20:11:40 +00:00

11 KiB
Raw Blame History

name, description
name description
post-draft-review Posts pull request review findings as GitHub draft (pending) inline comments for a human to edit and submit, instead of publishing them straight to the PR author. Use whenever asked to review a pull request or leave review comments on one, including phrasings like "review PR 764" or "take a look at this PR".

Post Draft Review Comments

A "draft review" on GitHub is a review left in the PENDING state. It is visible only to its author, accumulating comments, until the author explicitly submits (publishes) or discards it.

Findings go in as pending inline comments rather than individually published comments, so a human reviews them before anything reaches the PR author.

gh pr review can not create one — it always submits immediately (--approve / --comment / --request-changes). To stay in draft, hit the REST API: POST .../reviews with no event field creates the review PENDING rather than submitting it.

Inline comments only — the human authors the summary

Contribute inline comments only, created with an empty body (body: ""). The top-level summary is written by the human in the UI's "Finish your review" box when they submit. Rationale:

  • The high-level overview (praise, verdict, framing) is the human's voice, not the agent's.
  • The UI's summary box does not prefill an API-set body, so an agent-authored body is silently dropped on UI submit anyway, and can't be restored afterwards: PUT on a submitted review with an empty body is rejected with 422. An empty body sidesteps the whole failure mode.
  • Don't smuggle summary-level content into an inline comment. Top-level content anchored to a code line is poor UX for the reader.

Anything with no inline anchor — what was checked and found clean, issues in files outside the diff — goes in the chat hand-off, for the human to draw on when they write the summary.

Disclose that an agent found it

Each inline finding leads with :robot: followed by the bolded severity name and color (e.g. 🤖 **should-fix** 🟡 – …) to mark it as an AI finding. Keep it generic: don't name the agent, vendor, or model. Disclose only that an agent was involved. The human who finishes the review is accountable for what gets submitted.

Keep it tight

Bias toward brevity. A human reads this, then decides. Concretely:

  • Inline comments: lead with the severity tag and state the problem in a sentence or two, then the fix. Drop preamble, restated context, and pasted command/test output — point at the line and trust the reader.
  • Short sentences. One clause each where possible; no run-ons or long em-dash chains. State the claim and its consequence; leave the derivation (timelines, worked examples, step-by-step traces) out. The author can re-derive it from the claim, and can ask if they can't.
  • Severity tags: blocking 🔴 · should-fix 🟡 · nit 🟢 · question 🟢. Every inline finding opens with the robot marker, then the bolded severity name, then the color: 🤖 **blocking** 🔴 – …, 🤖 **should-fix** 🟡 – …, 🤖 **nit** 🟢 – …, 🤖 **question** 🟢 – …. Leading with 🤖 makes agent comments uniformly recognizable. The severity name is required because the colors alone aren't a standard PR authors know, and a bare 🟢 beside a critique reads like approval.
  • When in doubt, cut. A finding the reader can act on in five seconds beats a paragraph.
  • Brevity comes from cutting content, not compressing prose. Humans read these: keep complete sentences. No telegraphic fragments, abbreviations ("incl.", "w/"), or bare "label: clause" constructions. Drop whole points that don't change what the reader does next; write the surviving ones plainly.

Create the draft (PENDING) review

First check whether a pending review already exists. You post under the human's account, and each author may have only one pending review per PR — so any existing draft is theirs, and it may hold comments they wrote by hand:

gh api repos/<owner>/<repo>/pulls/<num>/reviews \
  --jq '.[] | select(.state == "PENDING") | {id, node_id}'

No output means there's no draft yet; create one as below. If it returns a review, skip to Adding to a draft that already exists and do not delete it.

Write each comment body to a file and assemble the payload with jq --rawfile, which escapes markdown, backticks, and newlines correctly. The body is always empty:

commit_id=$(gh pr view <num> --repo <owner>/<repo> --json headRefOid --jq '.headRefOid')

jq -n --arg commit "$commit_id" --rawfile c1 c1.md --rawfile c2 c2.md '{
  body: "",
  commit_id: $commit,
  comments: [
    {path:"cmd/ateapi/internal/store/ateredis/ateredis.go", line:849, side:"RIGHT", body:$c1},
    {path:"cmd/ateapi/main.go", line:57, side:"RIGHT", body:$c2}
  ]
}' > /tmp/review.json

gh api repos/<owner>/<repo>/pulls/<num>/reviews \
  --method POST --input /tmp/review.json --jq '{id, state}'
# => {"id": ..., "state": "PENDING"}

state: PENDING confirms it's a draft: not visible to anyone else, no notifications, until submitted. Keep the returned id to inspect or discard. Omitting event is what keeps it in draft; including event submits immediately.

Inline-comment notes:

  • line is the line number in the file at the PR head. side: "RIGHT" is the new (post-change) version, "LEFT" the base side.
  • Multi-line: use start_line and start_side together with line/side.
  • The line must fall within the PR's diff hunk, or the API rejects the whole review. A file that isn't in the diff at all (for example a caller that wasn't touched) can't take an inline comment. Anchor the point on a changed line nearby, such as the struct field that the untouched caller fails to populate, and reference the real location in prose. Otherwise leave it to the chat hand-off.
  • One bad anchor rejects the whole batch, and the error doesn't say which one. If the POST fails, create the review with a single comment and append the rest one at a time (see below) — the offender is then the one call that fails.
  • Pin exact head line numbers from a checkout of the PR head (gh pr checkout <num>, or fetch pull/<num>/head). Don't eyeball them from the diff.
  • commit_id pins the review to the head you actually read. Without it GitHub anchors against whatever is current when the POST lands, so a push between your fetch and your post silently moves every comment to lines you never looked at. The GraphQL append below has no equivalent field, so re-check the head SHA before appending to an older draft.

Adding to a draft that already exists

A pending review you didn't create is the human's, and it may already hold comments they wrote themselves. Never delete it. Deleting takes their comments with it, they aren't recoverable, and drafts are exactly where someone parks a half-finished thought.

Append instead, one mutation per finding, using the review's node_id (the PRR_… value, not the numeric id):

gh api graphql -f query='
mutation($review: ID!, $path: String!, $line: Int!, $body: String!) {
  addPullRequestReviewThread(input: {
    pullRequestReviewId: $review,
    path: $path, line: $line, side: RIGHT, body: $body
  }) { thread { id } }
}' -f review="PRR_kwDO..." \
   -f path="cmd/ateom-microvm/restore.go" -F line=214 \
   -F body=@c1.md

-F body=@c1.md reads the body straight from the file, keeping the file-per-comment discipline that --rawfile gives the batch path. Prefer it over -f body="$(cat c1.md)": command substitution strips trailing newlines, so the last line of the comment loses its break. For a multi-line anchor, add startLine and startSide.

Leave the review body alone whether it's empty or not. If the human started the draft, that text is theirs.

Appending this way is verified: the review keeps its id, stays pending, and the human's own comments come through untouched. The single-comment delete below hasn't been exercised — read the review back the first time you use it.

Revising your own findings

Delete your own comments one at a time. Don't delete the review — recreating it is only safe when you know the draft is entirely yours, and in a shared draft you don't:

# your findings are the 🤖-tagged ones
gh api repos/<owner>/<repo>/pulls/<num>/reviews/<id>/comments \
  --jq '.[] | select(.body | startswith("🤖")) | .id'

gh api repos/<owner>/<repo>/pulls/comments/<comment_id> --method DELETE

This is the second job the 🤖 marker does. In a shared draft it's the only thing that tells your comments apart from the human's.

A finding the human has already edited may no longer start with 🤖, so the filter skips it. That's the safe direction — they've taken it over — but don't read a missing marker as proof a comment was never yours.

Verify anchors landed correctly

line and original_line come back null while a review is PENDING (they resolve on submit), and position is a legacy cumulative diff offset. Neither tells you the file line at a glance. Instead, check the last line of each comment's diff_hunk, which is the line the comment is attached to:

gh api repos/<owner>/<repo>/pulls/<num>/reviews/<id>/comments \
  --jq '.[] | "\(.path)\n   ↳ \(.diff_hunk | split("\n") | last)\n"'

Hand-off, then the human finishes

The hand-off in chat should give the human what they need to write the summary and decide: a short findings table (severity, one-line claim, file:line anchor) and the checked-and-found-clean notes, meaning the verification work with no inline anchor. No paste-ready summary text — the summary is the human's to write.

Close the hand-off with the review id, the comment count, and these three points. A human who doesn't know them can publish the whole thing unread:

  • Nothing is published yet. The review is PENDING and visible only to them.
  • They should read each finding against the code before submitting, and delete anything they don't stand behind. Deleting a pending comment costs nothing.
  • Submitting publishes every remaining comment at once, under their name. There is no partial submit.

Point them at README.md, which is the full version of the above and the authoritative instructions for the human — keep it in sync if you change this section. Don't imply the review is finished. It isn't, and it can't be until someone has read the findings.

To discard the whole draft instead:

gh api repos/<owner>/<repo>/pulls/<num>/reviews/<review_id> --method DELETE

This removes every comment in the review, including any the human wrote. Only reach for it on a draft you created and know is entirely yours.

Gotchas

  • Each author may have only one pending review per PR at a time, and you post as the human, so a second POST .../reviews errors out. Append to the existing draft rather than clearing it — see Adding to a draft that already exists.
  • Relative markdown links in comment bodies (for example [x](cmd/.../foo.go#L1)) render oddly on GitHub. Prefer plain `path:line` in backticks for code references.