OWASP Finds ChatGPT Helped Low-Skilled User Take Over Admin Account in Hacking Test

OWASP found that ChatGPT helped an operator with no offensive security knowledge take over an administrator account during a controlled hacking test.

The operator largely copied commands, returned results to the chatbot, and followed its guidance to change a password and log in. The finding shows how AI can lower the skill needed for some attacks without removing the need for human input.

The LLM Exploit Generation Phase 2 report, published by the OWASP GenAI Security Project, compared human-guided chat sessions with autonomous agents against live Hack The Box targets.

These were authorized training environments, not breached businesses. Researchers tracked success, human effort, costs, safety controls, and failures.

The main test involved Silentium, a machine with a hidden staging website running a vulnerable version of Flowise. Using a testing guide generated by ChatGPT, the operator followed instructions with the common security tools nmap and curl.

ChatGPT directed information gathering and uncovered the staging site, which the operator would not have known to investigate. It then identified a login flow that revealed valid usernames and guided the operator through changing an administrator account’s password. The operator confirmed the takeover by signing in with the new credentials.

OWASP Finds ChatGPT-Assisted Admin Takeover

The result was not entirely skill-free. The test persona allowed general computer literacy, command-line use, and intermediate prompting ability. The operator also had to recognize refusals and reframe requests as authorized testing. Installing the Kali environment was outside the test’s scope.

Researchers also observed a response disappearing behind an error before a repeated request succeeded. They could not confirm the cause or reproduce the event, so it does not establish a general weakness in OpenAI’s safeguards.

The autonomous track tested nine configurations using CAI, HexStrike, Claude Code, OpenAI Codex, and hosted or local models. Stronger multi-agent setups discovered staging sites, identified vulnerable software, and researched exploit paths with little guidance after setup.

However, no autonomous run completed the final Silentium takeover step. Provider safety controls stopped the hosted systems that reached that stage, leaving humans to finish. Other runs stalled in repeated scans, password-cracking loops, hostname errors, or tool failures.

This distinction adds context to broader reporting on hackers using autonomous AI agents. In OWASP’s tests, the software around the model often mattered more than model choice, and building that software required expertise.

Costs also extended beyond API bills. Productive hosted runs cost tens of dollars, but setup, troubleshooting, and supervision remained important. A local Qwen Coder run avoided API charges yet needed repeated human corrections and ended with a tool integration error.

OWASP identified repeated, similar scans and requests against one target as a possible detection signal. Human-guided activity could be quieter, although the study did not directly measure stealth or detection rates.

For teams deploying agents, OWASP recommends enforcing approval before high-impact actions rather than relying only on model safeguards. Related coverage of hijacked AI pentesting agents highlights why those controls matter.

The report also warns against treating benchmark scores as proof of real-world hacking ability. Its small sample supports a narrower conclusion: AI speeds up parts of an attack, but reliable execution still depends on people.

Give your SOC the alert context to stop incidents quickly. Integrate Threat Intel from 16K SOCs.

The post OWASP Finds ChatGPT Helped Low-Skilled User Take Over Admin Account in Hacking Test appeared first on Cyber Security News.