Automating Azure DevOps PR Reviews with Claude Code

I shipped something small that has quietly improved how my team handles pull requests: an automated senior reviewer for Azure DevOps, driven by Claude Code. It catches real bugs, hands everyone back review time, and developers tell me it is right about 80% of the time - high enough that they act on its comments instead of scrolling past them.

The motivation was ordinary. Reviews are not hard, just constant: a dozen PRs a day, each wanting the same careful pass for the same handful of mistakes - a sequential await that should have been parallel, a filter that silently drops rows, a secret one import away from the client bundle. That is exactly the patient, never-bored work worth handing to a machine. The one thing that had to be true was trust: a reviewer only helps if people actually read it, so the whole design turns on a single rule - never post a confident guess. How it earns that trust is what this post is about, and it is the part that decides whether a review bot survives contact with real developers.

The official Azure DevOps MCP server runs only inside VS Code and Visual Studio, and I wanted this working unattended, not in my editor. So the approach is deliberately boring: a small script on my own machine, driving Claude Code locally and talking to Azure DevOps over its plain REST API with a personal access token. No new infrastructure - it runs where I already work.

The shape of it#

The bot is a small PowerShell engine plus a few prompt files that Claude Code reads. Once per tick it:

1. Polls Azure DevOps for active PRs on the branches I care about.
2. For any PR with new commits, pulls the diff (skipping lockfiles, generated code, and minified bundles so they never cost a token).
3. Runs a review prompt over the diff.
4. Posts each finding as its own resolvable thread, anchored to the exact file and line.

None of that is novel on its own. Two decisions are what made it usable.

Decision one: it verifies its own findings#

A language model reviewing a diff produces two kinds of finding: "this is broken" and "you might want to check that X still works." The second kind is where false positives live. The model cannot see the whole repository in a diff, so it hedges - "this export was removed; other code may depend on it" - and half the time the other code does not, in fact, depend on it.

So before posting anything, the bot tries to prove or disprove each finding against the actual source tree at the PR's head commit. It reformulates the finding as a testable hypothesis and checks it with git grep and git show against the fetched branch - never the local working copy, which is a subtle trap: your checkout is usually hours behind the PR you are reviewing.

A confirmed finding gets upgraded: instead of "this may break consumers," it posts "verified - src/foo.ts:42 calls this and will break." A disproved finding is dropped silently and remembered, so it never comes back on a later pass. This matters because a reviewer who gets burned by a few confident-but-wrong comments stops reading the rest - so checking every finding before it is posted is what keeps developers acting on the comments instead of tuning them out. When the team says it is right about 80% of the time, that number is the verifier talking; the raw prompt alone would be far noisier.

Decision two: a comment never closes a real bug#

The first version had a bug of its own, a social one. A developer would reply "good catch, fixing it," the bot would cheerfully resolve the thread, and the fix would never land - because "I'm fixing it" is a promise, not a diff. Resolved threads are invisible; the bug shipped.

So I gave a finding exactly two ways to close, and words are not one of them:

• The developer proves it wrong. An explicit ignore: <reason>, or a real technical justification ("this path only runs server-side"), resolves the thread as byDesign and records a signature so the same finding never returns on that PR. This is the only outcome that counts as a false positive.
• The code actually changes. A later re-review confirms the issue is gone from the source at the new commit, and only then closes the thread as fixed.

Everything else - agreement, thanks, "will fix", friendly chatter - leaves the thread open. If a developer pushes back without a reason, the bot counters exactly once with a concrete check they can run, then hands off to the human. It never argues twice. The point of the bot is to catch bugs, not to win comment threads.

The part I am quietly proud of: it costs nothing when idle#

Polling every ten minutes could mean waking an expensive model every ten minutes to conclude "nothing changed." Instead, the lane decision - is there a new commit, is there a new reply - is pure PowerShell hitting only the REST API. The model is invoked only when there is genuine work. On a quiet afternoon the poller runs for free; tokens are spent exactly when a human reviewer's attention would have been. I keep it running in a single terminal window:

while ($true) { .\.claude\skills\ado-pr-review\gate.ps1; Start-Sleep -Seconds 600 }

There is a Windows Task Scheduler version too, so it survives a laptop sleeping. And for a quick manual sweep inside a Claude Code session, it is just /pr-review.

What I would tell you before you run it#

• The review prompt is the whole game. Name your stack concretely and keep the categories narrow - bugs, security, performance, data integrity - with an explicit list of what to skip. The skip list matters as much as the report list.
• Start in dry-run. It logs everything and posts nothing. Watch it against real PRs for a few days, tune the prompt, then flip it live.
• It posts under your identity. The PAT is yours, so comments carry your name with an "automated review" footer. If that bothers the team, use a service account.
• Least privilege. When you create the PAT, give it exactly two scopes - Code (Read) and Pull Request Threads (Read & Write) - and nothing else. It never pushes commits or touches branches.

A net, not a substitute#

The bot is a fast, always-on net for a narrow set of high-value categories. It is not a stand-in for a careful review of a genuinely tricky change. For complex or high-risk PRs I still do a deep manual pass - these days with Matt Pocock's excellent code-review skill, run on demand. The two are complementary: the bot triages the everyday flow so my attention, and a heavier review, land where they actually matter.

Take it#

It is published as a Claude Code skill: claude-ado-pr-review. Copy the skill folder and the /pr-review command into any repo's .claude/, add a PAT and a small config file, and you have the same reviewer.

It works against Azure DevOps in the cloud today. On-premises Azure DevOps Server should work with config-only changes, because the engine is plain REST with the base URL read from config and no hardcoded host - but I have not had an on-prem instance to prove it against yet. If you run it there, I would genuinely like to hear how it goes.

One design note: the engine is AI-CLI-agnostic. It decides when there is work and hands a diff to an agent, and the review logic itself lives in plain-Markdown prompt files any capable agent can execute. The gate's spawn is now configurable - a command plus args in config.json, defaulting to Claude Code - so the same bot can drive Gemini CLI / Google Antigravity, GitHub Copilot CLI, or any headless agent that runs non-interactively. Only Claude Code is tested end to end so far; the rest are wired and waiting for a brave first run.

A closing thought: I would happily retire this. If Azure DevOps grows a native reviewer you can tag into a pull request - the way you can already bring Claude into a Slack thread or a GitHub PR - that would be a better home for this than a script on my laptop. The point was never the plumbing; it was the reviewer that verifies itself. Until that lands, this runs today.

The reviewer that never gets bored turned out to be worth building. The reviewer that verifies itself before speaking is the one the team actually kept.