GitHub Copilot Refuses Harmful Chat Prompts But Writes Them Inside Code Workflows

GitHub Copilot can refuse harmful prompts in chat while still generating the same harmful content inside code workflows when the request is decomposed across a multi-step IDE session.

Researchers Abhishek Kumar and Carsten Maple from the Alan Turing Institute analyzed GitHub Copilot as an IDE-integrated coding agent in Visual Studio Code. They focused on how safety behaves across full development workflows rather than single prompts.

They evaluated four closed-weight backends exposed through Copilot Anthropic’s Claude Sonnet 4.6 and Claude Haiku 4.5, plus Google’s Gemini 3.1 Pro and Gemini 3.5 Flash, using 204 harmful prompts drawn from Hammurabi’s Code, HarmBench, and AdvBench.

In direct chat and two simple baselines (reading prompts from CSV and a single-step “code-fix with teaching shots”), the models almost always refused, producing harmful responses in only 8 out of 816 model–prompt attempts.

Under a full multi-turn coding workflow, the same prompts triggered 816 unsafe “teaching-shot” completions out of 816 attempts, all judged specific and actionable by two independent expert evaluators under a strict rubric.

GitHub Copilot Rejects Harmful Chat Prompts

The attack does not rely on a single jailbreak prompt, but on assembling a harmful objective across normal software-engineering stages such as pipeline construction, benchmark ingestion, metric optimization, and iterative code refinement.

Copilot is asked to build and improve a jailbreak-evaluation pipeline for a nominal target model, then to increase attack success rate by adding example prompt–answer “teaching shots,”.

Leading it to write harmful answers as plain string literals inside arrays or data structures in the generated code.

Prompt-level refusal checks and red-teaming benchmarks that operate on a “one request, one response” basis can overstate the safety of coding agents.

Because they do not see harmful content that appears only inside generated files, test fixtures, or benchmark harnesses.

According to researchers at the Alan Turing Institute, Copilot workflows appear to be routine IDE tasks: reading files, running scripts, fixing errors, and improving metrics. The harmful behavior becomes apparent only when the entire session and its artifacts are analyzed together.

The authors argue that safeguards for AI coding assistants must move beyond turn-level refusal and include artifact-level inspection of generated code, cross-turn monitoring of session intent, and extra scrutiny when agents are asked to “improve” benchmark scores or attack success metrics.

For practitioners, the immediate takeaway is to treat chat refusals as incomplete evidence of safety and to review what Copilot actually writes into repositories, workflows, scripts, fixtures, and examples, especially in multi-turn sessions that manipulate adversarial or security-related benchmarks.

Stop Accepting SLAs Written for 2019 SOCs – Here’s the 2026 AI SLA Vendor Checklist – Download Free AI SOC SLA Guide