Hybrid Code Review Hits 81% on Real CVEs, Beats LLM Tools
Hybrid code review beats LLM tools with 81% accuracy on real CVEs. Discover why combining static analysis with AI is the winning security pattern.
AI coding agents made writing code nearly free. That single shift moved the bottleneck in software delivery. When a team can generate a feature in an afternoon that used to take a week, the constraint is no longer typing — it is trusting what was typed. Someone still has to confirm the code is correct, safe, and worth merging. That "someone" is increasingly another machine, and the market has spent the last year figuring out which machine does it best.
DeepSource (YC W20) recently launched Autofix Bot, a hybrid tool that pairs deterministic static analysis with a frontier AI agent. The pitch is direct: neither static-only tools nor LLM-only reviewers are good enough on their own, and combining them beats both. The benchmark numbers back the claim. This is worth unpacking, because the same architecture choice applies to almost any AI agent a business is about to deploy — the winning pattern is rarely "pure AI," it is "AI with a deterministic harness around it."
The bottleneck moved from writing code to reviewing it
Here is the pattern every engineering team is now living. Code volume is up, review capacity is flat. Human reviewers were already the slowest link before AI agents started opening pull requests at three in the morning. Now the queue is longer and the reviewer is more tired.
Automated review is the obvious answer, and two camps formed quickly. One camp bolts an LLM onto your pull requests and asks it to find bugs. The other runs classic static analysis — a fixed set of rules that scan your code for known bad patterns. Both are useful. Both, used alone, leave money on the table. Understanding why is the whole story.
Why static-only and LLM-only both fall short
Static analysis is precise but blind to context. Think of it as spell-check with a fixed dictionary. It is fast, it is deterministic — the same code produces the same findings every single time — and it never gets tired or distracted. The catch is the fixed dictionary. It flags patterns it was told to flag and misses everything nuanced or novel. Worse, it fires on things that are technically true but irrelevant in context, and that creates the single biggest reason teams abandon these tools: false-positive fatigue. Google's engineering team documented this precisely in their well-known writeup on building static analysis tools at scale — once developers stop trusting the alerts, they stop reading them, and the tool is dead weight no matter how clever the checker.
LLM review is smart but unreliable. An LLM reads code the way a sharp but distractible senior engineer would. It understands intent, catches logic errors a rule set never could, and explains itself in plain English. But it has four documented weaknesses that matter for a business:
- Non-determinism. Run the same review twice and you can get two different answers. That is a nightmare for a CI gate that is supposed to be a consistent checkpoint.
- Low recall on security. LLMs get "distracted" by style and readability and skate past the genuinely dangerous vulnerability. High-severity issues are exactly where you cannot afford a miss.
- Cost at scale. Feeding an entire diff plus surrounding context into a frontier model on every commit adds up fast across a busy repository.
- Attention drift. The more you ask it to look at, the more the important findings get diluted by trivial ones.
Put simply: static analysis is reliable but shallow, and LLMs are deep but flaky. The insight behind the hybrid approach is that each one's strength patches the other's weakness.
How the hybrid loop actually works
Autofix Bot runs three passes, and the design is worth studying because it is a template for building trustworthy AI agents in general.
Static pass: a high-precision baseline
The tool starts with 5,000+ deterministic checkers spanning code quality, security, and performance. This establishes a factual, repeatable baseline — a map of where the suspicious code lives before any LLM touches it. A sub-agent then suppresses context-specific false positives, addressing the exact fatigue problem that kills static tools in practice. This step is what turns a noisy rule engine into a clean signal.
AI review: static findings as anchors
Now the AI agent reviews the code — but it does not wander in blind. It uses the static findings as anchors, so its attention is pointed at the parts of the code that already look risky. Critically, the agent has real tools: access to the abstract syntax tree, data-flow graphs, control-flow, and import graphs — not just grep and a shell.
That distinction is the whole game. An agent limited to text search sees strings; an agent with a data-flow graph can trace how a value moves from user input to a dangerous sink. This is the same lesson we covered in why your AWS DevOps agent is only as good as its log: an agent's ceiling is set by the quality of the context and tools you hand it, not by the raw intelligence of the model. Give it structured graphs instead of raw text, and it reasons about code the way a compiler does rather than the way a search engine does.
Remediation: fixes validated before you see them
The third pass is where it earns the name. Sub-agents generate fixes, and a static harness validates every edit before emitting a clean git patch. In other words, the AI proposes and the deterministic layer verifies. You do not receive a plausible-looking suggestion that breaks the build — you receive a patch that has already passed the machine's own checks. That closed loop is what makes the output safe to trust at merge time.
The benchmark: what 81.2% actually means
DeepSource evaluated the tool on the OpenSSF CVE Benchmark, a public dataset of 200+ real JavaScript and TypeScript vulnerabilities drawn from actual CVEs. This matters because it is not a synthetic test — these are bugs that shipped and got exploited in the wild.
Real numbers:
| Tool | Accuracy | F1 |
|---|---|---|
| Autofix Bot (hybrid) | 81.2% | 80.0% |
| Cursor Bugbot | 74.5% | 77.42% |
| Claude Code | 71.5% | 62.99% |
| CodeRabbit | 59.4% | 36.19% |
| Semgrep CE (static-only) | 56.9% | ~3% |
Two things jump out. First, the hybrid leads on both metrics — accuracy and F1, which balances catching real issues (recall) against not crying wolf (precision). Second, look at the spread in F1. Accuracy alone can flatter a tool; F1 exposes it. Semgrep CE, the pure static-analysis entry, posts a respectable-looking 56.9% accuracy but an F1 that collapses into the low single digits — a signal that it is either missing most real vulnerabilities or drowning them in noise. The LLM-flavored tools land in the middle. The hybrid pulls clearly ahead of all of them.
Honest take: benchmarks are directional, not gospel. A single dataset in two languages does not settle the entire code-review market, and vendors choose the benchmarks that flatter them. But the OpenSSF set is public and built from real CVEs, and the gap here is wide enough — roughly seven accuracy points over the nearest competitor and a large F1 lead over the static-only baseline — that it points to something structural rather than a rounding artifact. The architecture, not luck, is doing the work.
Alternatives and honest trade-offs
No tool is free of cost, and choosing a code-review layer is a business decision, not just a technical one.
- Pure static analysis (Semgrep CE, linters) is cheap, fast, and fully deterministic. If your priority is a hard, repeatable CI gate for a known class of issues — and you can tolerate low recall on anything novel — it still has a place, often as one layer inside a bigger pipeline.
- Pure LLM review (Claude Code, CodeRabbit, Cursor Bugbot) shines at explanation and at catching intent-level bugs a rule set never anticipated. The trade-off is the non-determinism, the per-run cost, and weaker security recall shown above.
- Hybrid (Autofix Bot) wins on the benchmark but concentrates you on a newer product with a shorter public track record, and the security-narrowing step means you are trusting its false-positive suppression to be as good as advertised.
There is also a measurement question that outlives any single tool: how do you know your review layer is actually helping your team, not just a public benchmark? Merge times, escaped-bug rates, and how often developers accept versus dismiss the bot's suggestions are the signals that matter, and most of them are behavioral rather than stated. That is the same discipline we describe in reading the feedback users never type — the honest verdict on any agent lives in what people do with its output, not in the vendor's slide.
What this means for your project
The specific product is interesting, but the reusable lesson is bigger than code review. The winning architecture here is AI wrapped in a deterministic harness — a fast, cheap, repeatable layer that narrows the problem and validates the output, with the expensive, flexible model doing the reasoning in between. That pattern applies to document processing, financial reconciliation, support triage, and almost any agent a business is tempted to build as "just call the LLM."
Here is what we recommend for teams evaluating an AI review layer:
- Do not benchmark on demos — benchmark on your own repo. Run any candidate on a week of your real pull requests and measure acceptance rate and false positives against your codebase, not a public dataset.
- Treat determinism as a feature, not a nice-to-have. A CI gate that gives different answers on the same code erodes trust fast. Favor tools that pin the deterministic parts down.
- Judge the fix, not just the finding. A bot that flags a hundred issues but proposes no safe, build-passing patch just moves work around. The value is in validated remediation that a human can approve in seconds.
Key takeaway for business: the era of "pure AI does everything" is already giving way to "AI plus a verification layer that keeps it honest." Autofix Bot's benchmark lead is evidence for the general case — the highest-value AI systems pair a frontier model with a deterministic scaffold that catches its mistakes before they reach production. Whether you buy a code-review tool or build an internal agent, that is the architecture to insist on. Cheaper reasoning, higher recall on the issues that actually cost you money, and output you can trust enough to merge without a second full human pass.
This article is based on publicly available sources and may contain inaccuracies.


