Frequently asked questions

Short answers to the questions we hear most often. Each one links to a deeper page if you want the full picture.

What is a Review Credit (RC), and how many will my PR cost?

One RC ≈ 500 changed lines (additions + deletions, after binary/exclusion filtering, with long lines normalized at 200 chars). Empty diffs are free. A typical "medium PR" is ~1–2 RCs; that's where the "~75 medium PRs/month" framing on the pricing page comes from. See Plan tiers for the included RC count for each plan.

Learn more: Review credits explained

How do reviews actually appear on my pull request?

Two surfaces: (a) inline comments threaded to specific file:line ranges, each with a severity emoji + category + optional one-click suggestion code fence; (b) one PR-level summary comment that updates in place (queued → in progress → complete) and ends with a PASS, ADVISORY, or FAIL verdict. There's also a status check (ai-pr-review) you can wire up as a required branch policy.

Learn more: Anatomy of a review comment · Setting AI review as a required PR check

What happens when I run out of credits?

Depends on your plan's overage policy. Starter, Team, and Business default to auto-overage on both monthly and annual billing: reviews keep running once you pass your included RCs, and each extra RC is billed at the per-RC overage rate at period end (annual plans bill overage monthly via a metered add-on alongside the annual base). By default there is no cap; set an overage cap on the billing page if you want a known ceiling. Hard-block otherwise applies only to Enterprise or custom contracts (or when you set the overage cap to zero): new reviews are rejected — a failed status check is posted, but no summary comment or inline review threads appear — the dashboard's Reviews page shows the review with status Failed, and no credit is charged for the rejected review. Once your usage reaches your included RCs for the period — on hard-block or auto-overage alike — we send a one-time Review quota exhausted email to your account's billing contact for that billing period. Quota resets on your Stripe billing anniversary, not the 1st of the month.

Learn more: Review quota exceeded · Hard-block vs auto-overage

Do I have to install the Marketplace extension for reviews to run?

The extension is strongly recommended but not strictly required for reviews to run. ADO Pilot prefers its branded service-hook consumer (ms.vss-servicehooks.consumer contribution named "ADO Pilot"), which only registers in your organization when the extension is installed. If the extension is not installed, the onboarding wizard automatically falls back to Azure DevOps's built-in generic webhook consumer — PR events still reach the review pipeline, but the subscriptions appear under the generic "Web Hooks" entry on your Service Hooks page instead of the branded "ADO Pilot" entry.

The extension is still required for two capabilities that have no fallback: the "Run AI Review" entry in the PR action menu, and the organization-admin settings hub in Azure DevOps.

Learn more: Installing the Marketplace extension · Using the PR action menu

I installed the extension but haven't signed up yet — what now?

Installing the Marketplace extension alone doesn't turn on reviews. Until you sign up and connect your organization, the Settings Hub shows Not configured with a Get started card, and the PR action menu's "Run AI Review" returns Finish setting up ADO Pilot instead of starting a review — both link to signup. We also send the person who installed the extension a one-time "get started" email with the same link. Reviews begin automatically on new pull requests once you sign up and connect the organization.

Learn more: Quickstart · The organization admin settings hub

What permissions / scopes does the PAT need, and why?

ADO Pilot needs the following Azure DevOps personal access token (PAT) scopes. Pick exactly these — do not grant Full access.

ADO Pilot's onboarding wizard probes the Code scope at validation time. The Code Status and Pull Request Threads scopes are exercised when the wizard provisions service hook subscriptions and when ADO Pilot posts the first review. A PAT missing any of them will fail later in onboarding or at first review with a scope error — re-create the token with all three scopes selected if you see one.

If your PAT is missing the Code (Read) scope, the wizard catches it immediately and returns a pat_missing_scope error — but ADO's API doesn't tell us which scope failed, so the error just tells you to recreate the PAT with all three required scopes rather than naming the specific one. Missing Code (Status) or Pull Request Threads scopes aren't caught at this step; those surface later, at provisioning or on your first review.

Learn more: Creating a Personal Access Token · PAT and onboarding wizard error codes

What does ADO Pilot send to Anthropic, and is my code retained?

For each pull request, ADO Pilot sends Anthropic the diff content and filenames, structural context (like function and class signatures from the surrounding file), and any findings from our built-in static-analysis scanner — enough for a well-informed review without cloning or ingesting your full repo. Some deeper review modes also send the full contents of the changed files rather than just the diff hunks. Message content sent to Anthropic may be cached ephemerally as part of processing the request, but nothing is retained beyond that. On our side, the review findings, verdict, and usage metadata are kept in your account as your review history; the diff itself is deleted immediately once the review completes, with an automated safeguard guaranteeing deletion of any remaining copy within seven days at most — see the Privacy policy and DPA for the exact retention windows and access controls. If we need to re-review a PR, we always fetch the diff fresh from Azure DevOps rather than reusing a stored copy.

Learn more: Data handling — what we send to Anthropic · Privacy policy and DPA

Why do I sometimes see findings in the "in progress" comment that disappear from the final review?

That's the two-pass design working as intended. The first pass over-reports to maximize recall; the second pass re-evaluates each finding, drops false positives — often a meaningful share of the first pass's flags — and writes the final user-facing comments. Only the survivors are posted as inline threads.

Learn more: Why some findings are dropped before they reach your PR

What files does ADO Pilot skip, and how do I add my own exclusions?

Always skipped, no configuration needed: true binary files, plus a small set of built-in extensions like .lock and .map (source-map) files — this catches files such as Cargo.lock or yarn.lock, but not package-lock.json or pnpm-lock.yaml. Beyond that, exclusions are entirely up to you: fileExclusionPatterns starts empty for every org/project/repo, so add your own glob patterns — for generated code or minified bundles, for example — if you want them skipped. Excluded lines never count against your RCs. Add patterns in the dashboard under Repositories (per-repo). Filters are additive across the inheritance chain.

Learn more: Excluding files from review · What ADO Pilot won't review (and why)

A teammate updated the PR while a review was running — what happens?

ADO Pilot reviews a PR one iteration at a time and bills for each review it completes. A review that is already running finishes and is billed — though its results aren't posted, because a newer commit has superseded them. When several new iterations are waiting to be reviewed at once, ADO Pilot makes a best-effort attempt to review only the latest one and skip the earlier (superseded) iterations before they start, so those cost nothing. Either way, you'll see the summary comment update in place rather than two competing comments.

Learn more: Mid-review pushes — which commit gets reviewed?

How do I stop reviews running on a specific repo or branch?

Three knobs, narrowest first: at the repo level, toggle off the repo in the dashboard under Repositories; at the project level, set the branch filters to the branches you actually want reviewed (e.g., main, release/*); at the org level, you can remove all service-hook subscriptions, but that disables ADO Pilot for everyone. Status-check requirements set in ADO branch policies are independent — turning off reviews doesn't auto-remove the policy.

Learn more: Choosing which repositories get reviewed · Targeting branches