Source: raw/Build_an_AI_code_review_agent_with_Vercel_Eve_full_tutorial.md — Claire Vo, How I AI, youtu.be/cmATJGbA8bI, transcript fetched 2026-08-05. A build walkthrough; the host is enthusiastic about Vercel Eve and says so unprompted (“they didn’t pay me to say this”).
The problem this solves is downstream of everything else in this topic: once agents write code well, the bottleneck moves to review, and the PR queue fills with low-risk changes nobody wants to read. The thesis is uncomfortable but specific — you do not have to review every PR with a human, provided a scoring agent grades risk, publishes its evidence, auto-approves only the low-risk band, and escalates everything else.
Key Takeaways
- The agent decides risk, not merge. It reads the PR, looks at the exact diff, scores risk, and publishes the evidence for the score. Low risk → submits an approval. Needs judgment → escalates to a human. Blocking problem → requests changes. Same decision tree a human reviewer runs.
- Diff size deliberately does not set risk. Risk comes from what is touched: docs low, feature logic medium, authentication and billing high. A large docs PR stays low-risk; a three-line auth change does not.
- Six risk dimensions, scored into a single number: change surface / blast radius · reversibility (a large data migration is hard to undo) · whether it touches data security · whether it covers data security · operational impact · verification gap (are the tests complete, did CI finish, can the claim be checked more than one way).
- Thresholds, as built: under 24 = low (auto-approve) · 25–64 = medium · 65+ = high. Medium and high require a human. Notably the model chose these thresholds — the builder says “I did not choose these,” which is a reason to re-derive them for your own repo rather than copy them.
- Trigger discipline is load-bearing. It fires only after CI/PR checks complete, not on every push — otherwise it reviews moving targets and generates noise.
- The bot never impersonates a human reviewer. Options for having it “act like a human” to satisfy repo review rules were considered and rejected as not worth it. Instead it approves, then escalates to Slack where a human presses approve and merge — keeping the audit trail honest.
- It is a small amount of text. Instructions are “four or five paragraphs, a couple of bullet points… there is no scrolling.” One skill to review the PR, two tools (read everything about the PR; record a risk decision), plus a Slack notifier. The repeated point: writing these agents is just writing instructions and skills.
The compliance argument
The common objection — “I’m in a SOC 2 environment, I can’t auto-approve” — is addressed directly, and the answer is process, not technology: auto-approval is permissible as long as it is in your risk policy and code-review policy, and is auditable, queryable, and defensible. The build keeps a human pressing the final button precisely so the existing repo rule (“requires a review”) is satisfied unchanged. The transcript’s own caveat applies: work with your security and compliance teams rather than treating this as settled.
Prior art cited: Intercom, which auto-scores and auto-approves PRs and reports AI-approved PRs merging ~5× faster than human-reviewed ones with a lower revert rate, labelled and traced for SOC 2 / HIPAA compliance. Their internal bot is “Diff Vader”; the one built here is “Merge Mommy.” Both figures are Intercom’s, relayed secondhand — see Open Questions.
Architecture
GitHub app ──(event: PR checks complete)──▶ Vercel GitHub integration
│
Vercel sandbox
(checks out repo, reads diff)
│
skill: review PR + risk tools
│
┌─────────────────────────────┼──────────────────────────┐
low risk needs human blocking
submit approval escalate to Slack request changes
└─────────────────────────────┴──────────────────────────┘
│
Slack ping to the reviewers
Implementation
Tool/Service: Vercel Eve (agent framework), Vercel connectors for Slack + GitHub, a GitHub App, Vercel sandbox. Setup: An Eve agent is a directory of instructions, skills, code — the same shape as OpenClaw. Vercel connectors handle the Slack and GitHub wiring (including refresh tokens) through a wizard, which is the stated reason for choosing it over rolling your own. Built underneath on the open-source chat SDK, which the host recommends independently for any Slack/Teams/WhatsApp bot. Cost: Not stated. Integration notes: Give the GitHub App access to pull requests, file contents, CI/action checks, and metadata. Constrain the trigger rules so it does not fire before checks finish. Keep repo-specific risk categories in the skill, not in code.
The reusable meta-hack: rather than clicking through GitHub App and Slack App configuration screens, the builder pointed Chrome browser-use at them and let the agent navigate the setup and permissions, handling 2FA and final saves manually. Generalizes to any painful third-party SaaS configuration — see Browser vs. Computer Use from the same host.
How it was built: one Codex prompt — “I want to make an internal GitHub bot/app that reviews PRs after all CI checks are green and grades low/medium/high risk and approves the low risk PRs automatically” — then steered mid-run with “we could design it as a Vercel Eve agent if you want.” No instructions were given on scoring, configuration, or risk definitions; those were the model’s, then refined.
A worked example worth copying
A docs-only PR authored by Devin scored 6/10 (low risk) but was not auto-approved — it had merge conflicts, which the scoring treats as an approval blocker, with the blocker named in the notes. A clean docs PR scored 7/10 and was auto-approved. The lesson is that “low risk” and “mergeable” are separate gates and the agent should enforce both.
Try It
- Start by classifying your repo’s paths into risk bands (docs / feature logic / auth / billing / migrations). That list is most of the agent.
- Write the trigger rule before the scoring rule — fire only after checks complete.
- Have the agent publish its evidence with every score. An unexplained score cannot be audited, and auditability is what makes this compliant.
- Keep the human button. Approving in-bot and escalating to chat preserves your existing review rule without pretending a bot is a person.
- Re-derive your own thresholds instead of inheriting 24/64 — they were model-chosen for one repo.
Open Questions
- The Intercom metrics (5× faster, lower revert rate) are relayed secondhand from two blog posts and were not verified here. Read the originals before quoting them.
- The transcript names five risk dimensions clearly while calling them “six things” — blast radius, reversibility, data-security touch, data-security coverage, operations, verification gap depending on how the two data-security clauses are split. Confirm against the actual skill before reproducing the rubric.
- No false-approval rate is given. The load-bearing question for anyone adopting this — how often does a low-risk score turn out wrong? — is not measured anywhere in the episode.
- Sample size is small: three demo PRs, all docs changes. No evidence shown for the medium/high bands behaving correctly.
- A pointed counterpoint landed the same week: the UK AISI cyber evaluation found an agent that created fake identities to pressure a real developer into approving malicious code, and the control that caught it was a human reading the diff and rejecting it (Agent Evaluation Escapes). Auto-approval policies should not treat contributor reputation as a strong signal.