The New Hackers Are AI: Inside the Coming Wave of Offensive AI
This article is a synthesis of the ideas presented in "Chapter 6: AI Agents in Offensive Security," from the book "Agentic AI: Theories and Practices (Progress in IS)," authored by AI security researchers; Jerry Huang, Ken Huang, and
.
Imagine this: You get an urgent video call. It’s your company’s CFO, along with a few other senior executives you recognize. The connection is perfect, their faces and voices are familiar. They need you to authorize an emergency wire transfer for a secret, time-sensitive acquisition. It’s a huge sum, but the request seems legitimate. You make the transfer.
And just like that, your company is out $25.6 million.
This isn't a scene from a sci-fi movie. It happened to a multinational firm in Hong Kong last year. The employee was the only human on the call; everyone else was an AI-generated deepfake.
Welcome to the new frontier of cybersecurity. For years, we’ve thought of AI as a defensive shield—a smart system to detect threats and protect our networks. But a profound shift is underway. The most sophisticated offensive security tools are no longer being wielded by humans alone. The attackers—and the ethical hackers who hunt them—are becoming AI agents.
Based on groundbreaking research from security experts Jerry Huang, Ken Huang, and Chris Hughes, let's explore this emerging world of offensive AI. It's a domain where autonomous agents can probe defenses, deceive humans, and discover unknown vulnerabilities at a scale and speed we've never seen before.
The AI Red Team is Here
In cybersecurity, a "red team" is a group of ethical hackers paid to think like the enemy. They attack an organization's systems to find weaknesses before the real adversaries do. It's traditionally a painstaking, manual process requiring deep human expertise.
Now, the tech giants are automating their attackers.
Meta built GOAT (Generative Offensive Agent Tester), an AI designed specifically to "jailbreak" other AI models. It engages in strategic, multi-turn conversations to trick language models into violating their own safety rules. In one test, it successfully attacked Meta’s own Llama 3 model 97% of the time.
Google’s AI-Assisted Red Teaming (AART) framework uses AI to generate custom, culturally-aware attack scenarios. It pits "attack agents" against "evaluation agents" in a dynamic cat-and-mouse game to find vulnerabilities before they ever reach the public.
Microsoft released PyRIT, an open-source AI agent that not only hunts for traditional security flaws but also tests for "responsible AI" issues, like fairness and bias.
These aren't just simple scripts. They are autonomous agents that observe, plan, and act. They can analyze a system, devise a strategy, and launch thousands of unique, sophisticated attacks in the time it would take a human to grab a coffee. The human red teamer’s job is shifting from finding known bugs to discovering entirely new classes of risk that only these AI agents can uncover.
The Art of Deception, Perfected by AI
The Hong Kong deepfake incident is just one example of how AI is supercharging social engineering—the art of manipulating people into giving up confidential information. The classic phishing email with its tell-tale typos and awkward grammar is a thing of the past.
AI-Powered Phishing: Today’s AI can craft perfectly personalized phishing emails. It can scrape public data to learn about your job, your colleagues, and your recent projects, then use that information to create a message so convincing it's nearly impossible to spot. Imagine an email from your "boss" referencing a real conversation you had yesterday, asking you to click a link.
Voice Cloning (Vishing): With just a few seconds of audio from a YouTube video or a company conference call, AI can clone a person’s voice with terrifying accuracy. In one infamous case, criminals used a cloned director's voice to convince a bank manager to transfer $35 million. The manager was certain he was speaking to his boss.
The MGM Hack: While not explicitly confirmed to be AI, the massive 2023 cyberattack on MGM Resorts, which cost the company an estimated $100 million, began with a simple vishing call to the help desk. The attacker impersonated an employee to gain access. It's a chilling demonstration of how effective a simple, convincing voice can be.
The only defense against such sophisticated deception is a combination of technology and hyper-vigilance. Multi-factor authentication (MFA) is critical, but organizations must also establish strict verification procedures for sensitive requests—like confirming a transfer request over a completely different channel.
Poisoning the Well: AI in the Software Supply Chain
Every piece of modern software is built on layers of other software—open-source libraries, APIs, and development tools. This is the "software supply chain," and it's a prime target for a new breed of AI-driven attacks.
Instead of attacking a company's front door, AI can be used to sneak malware directly into the building blocks of its software. For example, an AI agent could:
Generate a Malicious Package: It could create a fake code library that looks like a useful tool (e.g., a "Simple Data Analyzer") and publish it online. When a developer innocently includes it in their project, the malicious code executes, stealing data or creating a backdoor. One recently discovered Trojan, BlankBot, disguised itself as a utility app on Android, using AI-like features to record keystrokes and steal banking credentials while cleverly evading antivirus detection.
Create a Fake API: The AI could intercept network traffic and redirect it to a fake endpoint that looks identical to a real one, allowing it to manipulate data or steal credentials in transit.
Infiltrate the CI/CD Pipeline: It could analyze a project's development pipeline, find a weakness, and automatically inject a tiny, malicious piece of code into the software right before it's deployed.
These attacks are incredibly stealthy. The AI can even analyze security tools and adapt its malicious code to remain hidden, making it a ghost in the machine.
The Automated Bug Hunters
The same AI capabilities can, of course, be used for good. The chapter's authors conceptualize two powerful AI agents for ethical hacking and vulnerability discovery.
First, imagine "BountyAgent," an AI partner for bug bounty hunters. Instead of manually reading hundreds of pages of rules for a program, the agent analyzes the scope, identifies promising areas, and recognizes complex vulnerability patterns that a human might miss. It can even help draft the final report, making the entire process faster and more effective.
Second, and even more powerfully, consider "DeepFuzz." "Fuzzing" is a technique where developers throw massive amounts of random data at a program to see if it crashes. It's effective but chaotic. DeepFuzz makes it intelligent.
Using deep learning, DeepFuzz doesn't throw random data; it learns what kind of data is most likely to break the target. It’s a guided missile for finding bugs. This isn't just theory—Google's "Big Sleep" AI, an evolution of this concept, recently discovered a brand new "zero-day" vulnerability in SQLite, one of the world's most widely used database engines. The AI found a flaw that had gone unnoticed by human researchers for years.
The Double-Edged Sword of AI
We are entering a cybersecurity arms race fought with algorithms. The same AI agents that companies like Google and Meta are building to secure their own platforms can be replicated by malicious actors. The ability to autonomously discover vulnerabilities, craft perfect deepfakes, and inject malware into software supply chains is a paradigm shift.
The future of security will be about fighting fire with fire—pitting defensive AIs against offensive AIs. This book chapter gives us a glimpse of the battlefield of tomorrow. By understanding the power of offensive AI, we can begin to build the resilient, intelligent defenses needed to survive it. The new hackers are here, and they're made of code. Are we ready? In the next article, we will introduce
Chapter 7:AI Agents in Defensive Security.
Stay tuned. If you can’t wait, please grab a copy at Amazon:
https://www.amazon.com/Agentic-AI-Theories-Practices-Progress/dp/3031900251
This book not only cover Agentic AI Security, but also dive very deep into business, economical and technical aspects of Agentic AI.