What happens when you social engineer an AI agent that was trained to be helpful over the phone? Can you get it to reveal its system prompt out loud? Will it disclose information about other callers? How far can you push it before its guardrails kick in?
AI voice agents sit behind telephony layers like speech-to-text, text-to-speech, and call routing that introduce new attack surfaces and opportunities. They’re replacing human operators everywhere: answering phones at doctors’ offices, handling IT help desks, triaging customer support, and booking appointments. They sound human, but underneath, they’re the same LLMs we’ve been prompt injecting.
I built an open-source tool that tackles this by placing real phone calls to voice AI agents, speaking attack scenarios using text-to-speech, capturing responses via speech recognition, and analyzing transcripts for signs of successful exploitation. It maps 20 attack scenarios across five categories from the OWASP Top 10 for LLM Applications: prompt injection, sensitive information disclosure, system prompt leakage, excessive agency, and misinformation. Detection uses pattern matching and an LLM judge to catch both obvious and subtle failures.

Willie Zhang

Willie Zhang is an Offensive Security Consultant with experience protecting companies by thinking like an attacker. What started as a $1 online course on ethical hacking in college turned into a career built on finding the gaps in systems that aren’t supposed to have any. Willie has a growing passion for understanding and attacking AI systems, building on a foundation of testing everything from corporate networks to the humans that run them. When he’s not learning something new, Willie is in a League of Legends lobby, because apparently cybersecurity isn’t chaotic enough.