What You Should Know Before Trusting AI Code Review: How Context Descriptions Affect Vulnerability Detection

Hello. This is Keito Inoshita from Affectosphere Group.

Imagine a DevSecOps review meeting. “We use an AI-powered code review tool, so our vulnerability detection is covered.” A reasonable statement — but is it the full story?

Here’s a question worth sitting with: when your AI code review tool flags a vulnerability, is it reading the code, or is it reading the context around the code?

A June 2026 study (Shahriar et al., arXiv:2606.30587) ran a controlled experiment across 3 programming languages and 8 LLMs. The code stayed the same. Only the context — commit messages, variable names, function names, comments — changed. The detection results did not stay the same.

---

Three things this article covers

• All 8 LLMs tested were susceptible to three cognitive heuristics: halo effect, framing effect, and anchoring. Changing only contextual descriptions — without touching the code — altered vulnerability detection outcomes.
• The framing effect showed the largest average susceptibility at 33.2%, meaning that roughly 1 in 3 detections can be influenced by how the code’s context is framed.
• Vulnerabilities requiring semantic understanding were more susceptible to cognitive influence than those detectable through pattern recognition — which means the most serious vulnerabilities are most at risk of being missed.

---

① The experimental design: same code, different words

What makes this study methodologically rigorous is its control structure.

The code was identical across conditions. Vulnerable code stayed vulnerable; safe code stayed safe. What changed was only the surrounding language: commit messages, function names, variable names, inline comments.

The halo effect condition, for example, measured whether attaching a positive contextual label — “reviewed by security expert,” “production-grade implementation” — to genuinely vulnerable code changed the LLM’s detection verdict.

Three programming languages (C, Python, JavaScript) and 8 LLMs. A large-scale controlled experiment with a clean manipulation.

---

② Halo effect: “looks good” becomes “probably safe”

The halo effect is a well-known cognitive bias: one positive impression bleeds into overall judgment. The study found LLMs exhibit the same pattern.

When vulnerable code came with contextual cues like “security-reviewed” or “written by a senior engineer,” the probability of the LLM classifying it as safe increased.

In practice: a PR description that says “refactored and tested” or “passing all security checks” may cause an AI reviewer to de-prioritize its scan. Clean variable naming, organized comments, structured commit messages — all of these can produce a “this looks like careful code” impression that reduces scrutiny.

---

③ Framing effect: 33.2% average susceptibility

The framing effect was the strongest of the three heuristics, with an average susceptibility of 33.2% across models.

The framing effect describes how the same information, presented differently, yields different judgments. In code context: “this function skips input validation for performance reasons” versus “this function omits input validation” — same code, different framing, different verdict.

The first framing implies a deliberate design choice. The second implies a potential problem. The code is identical.

At 33.2%, this is not a minor noise effect. In more than 1 in 3 cases, how a developer describes their code in a commit message may determine whether an AI tool flags a vulnerability or passes it.

---

④ Anchoring and the semantic vulnerability problem

Anchoring occurs when early information biases subsequent judgments. A positive anchor — seeing favorable context first — makes it harder to flag problems detected later.

The study also identified a structural asymmetry worth noting. Vulnerabilities detectable through pattern recognition (SQL injection, classic buffer overflows) showed lower susceptibility to cognitive heuristics. Vulnerabilities requiring semantic understanding — logic errors, authentication flow defects, context-dependent access control issues — showed higher susceptibility.

This creates a troubling hierarchy: the vulnerabilities LLMs are already good at catching are the ones least affected by cognitive bias. The vulnerabilities that require deeper understanding — and that tend to carry higher security impact — are the ones most susceptible to being missed when context is misleading.

---

⑤ Implementation recommendations for DevSecOps and security audit vendors

The practical takeaways from this research are clear.

Standardize context input. Establish guidelines for what gets passed to an AI code review tool. Strip emotionally or evaluatively loaded language from commit messages, function descriptions, and inline comments before they reach the model. A preprocessing step that removes phrases like “security-reviewed,” “refactored,” or “already tested” can reduce the surface area for halo and framing effects.

Introduce validation gates. For high-criticality code paths, scan the same code under multiple context variants and require consistent detection across variants before passing. Any code where the LLM verdict changes based on contextual framing alone should be flagged for mandatory human review.

Define a consistency KPI. Measure “context consistency score” — the agreement rate in LLM verdicts when the same code is scanned with different contextual framing. Track this metric per tool and per model version to guide tool selection and operational improvement.

For security audit tool vendors, susceptibility to context manipulation is a product differentiator. Including cognitive heuristic resistance in your benchmark suite offers a measurable quality dimension that traditional detection-rate metrics miss.

---

Conclusion: separating what the AI reads from what it should read

LLM-based code review tools are not reading code in isolation. They are reading code within a context — variable names, commit messages, comments — and that context influences their judgments in measurable, non-trivial ways.

The framing effect at 33.2% average susceptibility is not a footnote. It is a structural risk that any organization relying on AI code review needs to address operationally.

Using AI for security review and trusting AI for security review are different decisions. This research quantifies the gap between them.

---

Reference

1. Shahriar, Asif, Cai, Hongyu, Benkraouda, Hadjer, Wang, Gang, & Celik, Z. Berkay. (2026). Words Speak Louder Than Code: Investigating Cognitive Heuristics in LLM-Based Code Vulnerability Detection. arXiv preprint (2026).

* This article was written in part with AI assistance and may contain inaccuracies.